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Foreword 


This  report  was  developed  by  Shantanu  Dhar  and  Robert  S.  Matthews  of  the  Industrial 
Technology  Institute  (ITI)  under  a National  Institute  of  Standards  and  Technology 
(NIST)  task  order  contract  number  43NANB020484.  ITI  is  a not-for-profit  research 
and  development  organization  located  in  Michigan,  whose  mission  is  to  advance  the 
competitiveness  of  American  manufacturing  organizations.  ITI  technical  staff  is 
experienced  and  educated  in  manufacturing-related  technologies  and  organizational, 
economic,  and  business  issues.  One  area  (of  many)  where  ITI  and  NIST  interests 
coincide  is  in  the  development  and  conformance  testing  of  STEP.  NIST  published  the 
National  PDES  Testbed  Development  Plan  for  a STEP  Conformance  Testing  Service 
(NISTIR  4641),  which  defines  an  important  need  for  the  conformance  testing  (CT)  of 
STEP  products. 

The  work  described  in  this  document  was  funded  by  the  U.S.  Government’s 
Department  of  Defense  Computer-aided  Acquisition  and  Logistic  Support  (CALS) 
initiative.  Although  the  CALS  Evaluation  and  Integration  Office  and  NIST  do  not 
necessarily  endorse  the  recommendations  found  here,  it  is  important  to  make  public 
such  recommendations  for  discussion.  This  report  focuses  on  important  issues  that  will 
arise  during  an  effort  to  offer  a full-scale  STEP  conformance  testing  service  to  U.S. 
industry.  The  content  presented  here  draws  fi’om  ITI’s  past  conformance  testing 
experience  in  order  to  provide  a perspective  of  the  real-life  problems  associated  with 
developing  and  offering  testing  services.  Thus,  to  the  decision-maker  and  funding 
agency  considering  active  participation  in  STEP  CT,  this  report  identifies  major  issues 
confronting  CT  in  general,  and  STEP  CT  in  the  United  States  in  particular.  It  also 
offers  insight  into  the  direction  that  U.S.  activity  should  proceed. 

This  report  is  not  subject  to  copyright. 
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Executive  Summary 


The  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  is  an  emerging  set  of 
standards  that  will  revolutionize  the  use  and  exchange  of  product  information  by 
manufacturing  organizations.  In  order  for  STEP  to  achieve  this  objective  of  free 
exchange  and  use  of  product  data,  STEP-conforming  products  based  on  standardized 
application  protocols  have  to  be  available  to  users.  Moreover,  since  such  products  will 
interact  and  transmit  information  among  each  other,  it  is  imperative  that  they 
"interoperate"  (work  together  or  share  data  appropriately). 

The  National  Institute  of  Standards  and  Technology  (NIST)  actively  participates  in  the 
STEP  development  effort  to  support  U.S.  industry.  NIST  and  other  U.S.  activities 
contributing  to  STEP  are  known  as  PDES  (Product  Data  Exchange  using  STEP).  NIST 
has  developed  an  overall  strategic  plan  for  the  National  PDES  Testbed,*  along  with  an 
associated  long-range  development  plan  for  Conformance  Testing^.  The  objective  of 
this  activity  is  to  enable  the  broad-based  adoption  of  STEP-conforming  products. 

The  objective  of  this  report  is  to  evaluate  the  need  for  a conformance  testing  (CT) 
service  for  STEP  implementations,  and  identify  the  steps  that  should  be  taken  to 
develop  an  appropriate  CT  service.  Two  important  assumptions  underlie  the 
conclusions  presented  in  this  report.  First,  experiences  gained  from  CT  services  of 
other  standards  implementations  can  be  adapted  to  STEP.  Second,  STEP  is  not  a 
single  standard  but  a collection  of  standards.  Among  this  collection  are  application 
protocol  (AP)  standards,  on  which  conformance  testing  of  implementations  will  be 
based.  An  application  protocol  defines  the  context  for  the  use  of  product  data  and 
specifies  the  use  of  the  standard  in  that  context  to  satisfy  an  industrial  need. 

This  report  is  deliberately  freestanding  and  fairly  generic.  Many  of  the  issues  raised 
here  have  been  or  are  being  considered  by  ISO  TC184/SC4,^  the  international 
organization  responsible  for  STEP  standardization. 


‘Charles  McLean,  National  PDES  Testbed  Strategic  Plan.  NISTIR  4438,  September  1990. 

^ Sharon  Kemmerer,  National  PDES  Testbed  Development  Plan:  STEP  Conformance  Testing 
Service.  NISTIR  4641,  August  1991. 

^International  Organization  for  Standardization  (ISO)  Technical  Committee  on  Industrial 
Automation  (TCI  84)  Sub-Committee  on  Industrial  Data  and  Global  Manufacturing  Programming 
Languages  (SC4). 
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Conformance  testing  for  standards  such  as  STEP  is  a means  of  increasing  the  likelihood 
that  two  different  systems  operating  under  the  same  standard  will  work  together.  It  is 
based  on  the  concept  that  evaluating  implementations  of  an  AP  against  the  standard 
itself,  then  giving  the  vendor  time  to  make  any  corrections,  will  eliminate  most 
obstacles  to  successful  interoperability. 

When  implementations  of  standards  for  information  representation  and  exchange  were 
few  in  number,  the  evaluation  of  products  required  only  the  straightforward  operation 
of  determining  whether  they  interoperate.  Testing  every  product  with  every  other 
through  interoperability  explodes  to  a complex  task  as  more  products  exist.  This  is 
why  conformance  testing  (CT)  is  the  most  important  method  of  evaluating  a product’s 
adherence  to  a standard.  Conformance  testing  does  not  remove  the  need  for 
interoperability  testing,  but  it  can  dramatically  reduce  the  amount  of  effort  required  to 
get  two  differing  systems  to  work  together.  The  cost  of  CT  is  not  insignificant,  but  if 
executed  properly,  it  reduces  the  overall  cost  of  getting  systems  to  interoperate. 

The  four  main  participants  concerned  with  CT  are  the  standard  bodies  that  develop  the 
standards,  the  vendors  who  build  standard-conforming  products,  the  users  who  require 
these  products  and  need  the  assurance  the  products  will  interoperate,  and  the  testing 
community  which  includes:  testing  laboratories,  accreditation  authorities,  and 
certification  bodies.  To  be  successful,  CT  needs  to  satisfy  the  needs  of  all  these 
entities.  It  must  reflect  the  true  intent  of  the  standard  in  question,  be  inexpensive  in 
the  long  run,  and  build  the  users’  confidence  in  the  value  CT  provides. 

There  are  some  strategic  issues  to  consider  when  planning  for  STEP  CT.  The 
involvement  of  U.S.  bodies,  particularly  the  Department  of  Defense  (DoD),  is  critical. 
Since  DoD  and  its  suppliers  will  be  major  users  of  STEP,  the  indigenous  development 
of  CT  capabilities  is  essential.  Another  strategic  issue  is  that  of  CT  system  openness. 
Once  developed,  STEP  CT  systems  should  be  publicly  available  to  vendors  and  users 
to  perform  in-house  testing  to  help  in  product  and  application  development. 

A large  number  of  procedural  and  administrative  issues  are  associated  with  offering 
STEP  CT.  To  begin  with,  a single  or  small  number  of  STEP  APs  should  be  used  to 
evaluate  CT  services.  Then  the  experience  gained  in  this  preliminary  exercise  should 
be  applied  toward  additional  APs.  Since  funding  is  an  important  factor,  major 
beneficiaries  of  these  services  would  be  expected  to  fund  a CT  development  activity. 

A number  of  recommendations  are  made  as  a result  of  this  study.  Among  the  most 
important  are: 
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U.  S.  participants  must  have  a mechanism  for  providing  requirements  into 
their  STEP  CT  system. 

The  United  States  may  need  to  create  its  own  STEP  CT  system  so  as  to  not 
rely  on  other  countries. 

The  CT  system  itself  should  be  made  widely  available,  and  in-house  testing 
should  be  encouraged  prior  to  formally-supervised  CT. 

CT  system  development  should  start  small,  both  in  number  of  application 
protocols  addressed,  and  in  the  facility  established  (resources  and  equipment) 
to  undertake  development. 

CT  systems  should  be  made  highly  automated  for  low-cost  operation  and 
high  accuracy. 
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1.  Introduction 


The  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  is  an  emerging  set  of 
standards  that  will  revolutionize  the  use  and  exchange  of  product  information  by 
manufacturing  organizations.  In  order  for  STEP  to  achieve  this  objective  of  free 
exchange  and  use  of  product  data,  STEP-conforming  products  based  on  standardized 
application  protocols  have  to  be  available  to  users.  Moreover,  since  such  products  will 
interact  and  transmit  information  between  each  other,  it  is  imperative  that  they 
"interoperate"  (work  together  or  share  data  appropriately). 

Conformance  testing  is  the  evaluation  of  a product  to  see  whether  it  meets  a particular 
standard.  Conformance  testing  (CT)  for  the  products  based  on  technologies  like  STEP 
(Standard  for  the  Exchange  of  Product  Model  Data),  where  many  suppliers  and 
products  must  interoperate,  is  critical  to  STEP’S  rapid  adoption  by  implementors  and 
users.  CT  increases  the  user’s  confidence  that  the  products  will  work.  The  initial 
planning,  organizational  interfacing,  and  prototyping  of  a CT  system  will  be  most 
effective  if  coordinated  with  STEP  standards  development  activities.  These  activities 
include  those  associated  with  application  protocol  prototyping  and  validation,  STEP 
standardization  activities,  and  early  commercial  product  development.  In  particular,  AP 
validation  activities  can  contribute  test  data  and  software  to  the  development  of  CT. 

The  objective  of  this  report  is  to  evaluate  the  need  for  a conformance  testing  (CT) 
service  for  STEP  implementations,  and  identify  the  steps  that  should  be  taken  to 
develop  an  appropriate  CT  service.  Two  important  assumptions  underlie  the 
conclusions  presented  in  this  report.  First,  experiences  gained  from  CT  services  of 
other  standards’  implementations  can  be  adapted  to  STEP.  Second,  STEP  is  not  a 
single  standard  but  a collection  of  standards.  Among  this  collection  are  application 
protocol  (AP)  standards,  on  which  conformance  testing  of  implementations  will  be 
based.  An  application  protocol  defines  the  context  for  the  use  of  product  data  and 
specifies  the  use  of  the  standard  in  that  context  to  satisfy  an  industrial  need. 

The  following  philosophy  should  guide  CT  development  efforts.  Foremost,  CT  and 
associated  tools  must  help,  not  hinder,  market  adoption  of  conformance-tested  products. 
CT  development  efforts  can  lose  sight  of  this  objective  if  the  developers  are  not  careful. 
Tools  or  services  which  perform  to  CT  specifications  could  still  be  technically 
cumbersome,  too  expensive,  or  politically  unacceptable  because  of  a lack  of  consensus. 
To  overcome  such  pitfalls,  it  is  key  for  CT  developers  to  regularly  check  decisions 
from  the  perspective  of  how  will  this  help  market  adoption  and  stimulate  the 
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availability  of  useful  products.  By  bearing  this  in  mind,  CT  will  stimulate  increasing 
buyer  confidence  in  STEP  standards  and  STEP-based  products. 


1.1  Organization  of  this  Report 

In  addition  to  this  introduction,  this  report  has  four  major  sections.  Recommendations 
on  a specific  issue  are  made,  as  appropriate,  in  the  subsection  dealing  with  that  issue. 

• Section  2 identifies  CT  issues  and  makes  recommendations  for  dealing  with 
such  issues  for  STEP. 

• Section  3 discusses  evaluation  procedures  which  include  issues  relating  to 
the  abstract  and  executable  test  suites  (ATSs  and  ETSs)  and  tools,  and 
evaluation  of  CT  systems  and  testing  laboratories. 

• Section  4 proposes  a set  of  milestones  for  the  realization  of  STEP  CT 
services  in  the  United  States. 

• Section  5 concludes  the  report  with  a summary  of  the  recommendations. 


1.2  Definitions  and  References 

Those  terms  most  fi-equently  used  in  this  report  are  defined  in  Section  6.  Unless 
otherwise  noted,  the  definitions  are  extracted  fi’om  ISO  CD  10303-31,  "Conformance 
Testing  Methodology  and  Framework,  General  Concepts,"  of  15  November  1990. 

In  addition  to  ISO  CD  10303-31,  the  following  documents  have  been  used  in  the 
preparation  of  this  report: 

ISO  DIS  9646,  OSI  Conformance  Testing  Methodology  and  Framework,  Parts  1,2, 
4,  and  5.  [Available  through  the  ISO  Secretariat:  1,  rue  de  Varembe,  Case  postale 
56,  CH-1211  Geneva  20,  Switzerland.] 

ISO/IEC  Guide  25,  General  Requirements  for  the  Technical  Competence  of 
Testing  Laboratories.  [Available  tlurough  the  ISO  Secretariat:  1,  rue  de  Varembe, 
Case  postale  56,  CH-1211  Geneva  20,  Switzerland.] 
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ISO  WD  10303-1,  "STEP  Overview  and  Fundamental  Principles,"  August  1990. 
[Available  through  ISO  TC184/SC4  Secretariat:  NIST,  220,  A127,  Gaithersburg, 
MD  20899,  USA.] 

ITI  Technical  Report  87-14.1,  Test  Coverage  Analysis  and  Measurement  (TCAM) 
- A Practical  Approach  to  Determining  Coverage.  [Available  through  ITI:  P.O. 
Box  1485,  Ann  Arbor,  MI  48106,  USA.] 
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2.  Conformance  Testing  Issues  and  Recommendations 


This  section  discusses  various  issues  relevant  to  STEP  CT.  The  approach  and  structure 
of  this  report  have  been  determined  through  discussions  between  NIST’s  contract 
representatives  and  ITI  about  the  issues  that  need  to  be  addressed.  These  discussions 
have  led  to  the  following  major  areas  of  concern: 

• High-Level  Issues 

• Strategic  Issues 

• Development  Issues 

This  section  is  an  examination  of  the  nature  of  CT  and  its  characteristics  and  how  they 
affect  the  task  of  developing  and  offering  CT  services.  Section  2.1  presents  these  ideas 
for  CT  in  general,  and  STEP  CT  in  particular. 

In  any  standard-related  activity,  be  it  standard  development,  implementation,  or  CT, 
there  are  four  main  participants  that  are  primarily  involved.  These  are  the  standards 
bodies  which  develop  the  standards,  the  developers  who  will  build  standard-conforming 
products,  the  users  who  will  use  these  products,  and,  last  but  not  least,  the  testing 
community.  Of  these  four  main  participants  each  has  its  own  concerns  and  needs  and 
has  to  position  itself  strategically  in  order  to  fill  these  needs.  Section  2.2  discusses 
these  strategic  issues. 

Developmental  concerns  are  an  important  component  for  the  task  of  developing, 
offering,  and  arbitrating  CT.  Standards  bodies  and  user  groups  (who  are  interested  in 
seeing  the  standard  get  widespread  acceptance)  are  concerned  with  the  availability  of 
high-quality  standards,  CT  facihties,  and  competent  CT  staff.  Developers  want  to 
ensure  CT  costs  are  reasonable,  and  CT  is  conducted  fairly  and  without  prejudice.  Past 
experience  has  shown  that  an  efficient  and  cost-effective  way  to  conduct  CT  for 
networking  protocols  is  with  a set  of  tools  that  are  modular,  configurable,  extendable, 
and  automated.  For  STEP  this  is  particularly  significant.  The  right  tools  that  can  be 
configured  to  a given  ATS,  will  automate  the  development  of  executable  test  cases 
(ETCs)  and  save  on  resources  that  would  have  been  expended  in  developing  them 
manually.  Section  2.3  deals  with  these  issues  and  lists  brief  descriptions  of  paper-and- 
pencil  tools,  software,  and  hardware  that  could  be  used  to  automate  the  CT  process. 
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2,1  High-Level  Issues 


Conformance  testing  often  means  different  things  to  different  people.  Because  of  this 
confusion  there  is  a need  to  define  CT,  discuss  its  limitations,  and  describe  measures 
of  success.  On  some  occasions  CT  has  been  promoted  as  the  cure  for  incompatibility, 
which  it  is  not.  On  the  other  hand,  developers  may  see  it  as  an  extra  hurdle  in  the  path 
to  commercialization.  The  following  subsections  identify  some  important  issues  in 
STEP  CT  and  provide  the  basis  for  the  discussions  on  strategic,  developmental,  and 
procedural  issues  that  follow. 


2,1.1  Scope  of  Conformance  Testing 

Successfully  completing  the  conformance  assessment  process  is  not  a guarantee  of 
interoperability.  CT  may  discover  errors  in  an  implementation;  it  cannot  however 
demonstrate  the  implementation  is  error  free.  For  a moment,  let  us  assume  that  all 
conformance  faults  have  been  eliminated  for  a particular  implementation.  This  would 
mean  that  an  implementation  would  faithfully  follow  its  specified  behavior  and  that 
behavior  required  by  the  standard.  But  standards  have  their  own  inherent  (and 
sometimes  unavoidable)  weaknesses.  Validation  and  verification  of  STEP  APs  will 
hopefully  eliminate  the  most  serious  problems  of  this  nature  prior  to  standardization, 
and  thus  prior  to  CT.  Remaining  subtleties,  faults,  ambiguities,  or  omissions  in  a 
standard  can  lead  to  two  conforming  implementations  failing  to  interoperate  in 
executing  a desired  function.  Thus,  demonstration  of  conformance  does  not 
unequivocally  translate  to  a guarantee  of  interoperability.  The  standards  development 
process  must  balance  two  diametrically  opposed  concepts:  (1)  over-specifying 
requirements  to  guarantee  interoperability;  or  (2)  under-specifying  requirements  to 
provide  maximum  latitude  to  implementors  to  optimize  performance  with  respect  to 
cost 

Conformance  testing  can  be  used  to  help  in  debugging  implementations.  ETSs  and 
testing  tools  developed  for  testing  laboratories  could  be  offered  to  help  developers  in 
such  debugging.  Also,  CT  system  openness  (see  Section  2.2.2)  can  allow  developers 
to  debug  implementations  at  their  premises  without  using  accredited  CT  facilities. 
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2d, 2 Why  Conformance  Testing  is  Useful 

The  four  main  participants  concerned  with  CT  have  different  views  of,  and  expectations 
from,  CT.  From  the  perspective  of  the  standards  bodies,  CT  is  key  to  facilitating  the 
acceptance  and  adoption  of  standards.  Without  CT  a standard  exists  only  on  paper,  and 
vendors’  claims  of  conformance  cannot  be  validated.  In  a sense,  the  conformance  test 
is  the  standard.  Further,  CT  can  provide  additional  validation  of  the  standard  itself. 
Even  though  an  AP  was  tested  before  being  established  as  a standard,  CT  may  yet 
expose  parts  of  a standard  that  are  difficult  to  implement,  contradictory  in  nature,  or 
too  complex.  Thus  CT  can  provide  valuable  feedback  to  standards  bodies  to  help  them 
modify  standards  appropriately  and  thereby  facilitate  acceptance  and  adoption  among 
vendors  and  users. 

From  the  users’  point  of  view,  CT  gives  them  confidence  in  a product  in  much  the 
same  way  as  products  tested  and  marked  by  the  Underwriters  Laboratory~the  product 
does  what  it  is  intended  to  do.  In  the  long  run,  CT  aims  directly  at  reducing  costs 
associated  with  product  acquisition,  operation,  and  maintenance. 

In  order  to  build  conformance  products,  developers  must  invest  in  detailed  and  careful 
design,  test  thoroughly,  and  subject  the  product  to  CT.  Thus,  developers  often  see  CT 
as  an  impediment  in  their  ability  to  quickly  respond  to  market  needs,  and  as  a factor 
that  raises  product  prices.  However,  more  and  more  developers  agree  that  if  done  right, 
an  early  start  in  CT  makes  the  task  of  developing  standard-conforming  products  easier, 
less  expensive  in  the  long  run,  provides  a marketing  advantage,  and  gives  users 
confidence  in  these  products. 

CT  services  and  tools  should  be  ready  when  STEP  is  accepted  as  an  international 
standard.  The  Initial  Graphics  Exchange  Specification  (IGES)  is  an  example  where 
formal  conformance  testing  requirements,  tools,  and  procedures  for  a standard  have  not 
achieved  acceptance  within  the  U.S.'^  but  many  implementations  exist.  Having  no 
conformance  requirements  against  which  to  gauge  developmental  requirements  has 
proven  costly  to  the  vendor,  and  therefore  costly  to  the  user.  In  addition,  users  are 


^ Since  this  report  was  written,  an  IGES  CT  system  and  service  developed  by  the  CAD- 
CAM  Data  Exchange  Technical  Centre  (CADDETC)  has  been  accredited  by  the  British  National 
Association  of  Measurement  Accreditation  Services  (NAMAS).  Mumal  recognition  has  been 
established  with  four  other  European  countries.  Alternatives  for  U.S.  IGES  CT  include  licensing 
the  CADDETC  CT  system  for  use  in  the  U.S.  or  U.S.  recognition  of  NAMAS  accreditation. 
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often  reluctant  to  accept  vendor  claims  of  product  conformance  and  therefore  marketing 
of  the  product  is  delayed. 


2d 3 Conformance  Versus  Interoperability  Testing 

Traditionally,  for  standards  involving  the  interaction  of  different  companies’  products, 
pairs  of  vendors’  implementations  were  tested  to  see  if  they  interoperate.  This  process 
is  known  as  interoperability  testing.  To  correct  problems  with  the  two  systems’ 
interactions,  modifications  were  then  made  to  either  product  (or  both)  based  on  what 
was  convenient.  Thus,  a de  facto  standard  evolved  as  a few  key  developers  iteratively 
developed,  tried  to  interoperate,  went  back  to  the  drawing  board  to  modify  their 
implementation,  and  documented  the  assumptions  based  on  their  interactions.  This 
approach  was  acceptable  for  three  reasons:  there  were  relatively  few  standards,  few 
interested  developers,  and  no  better  method  was  known. 

Today,  standards  exist  or  are  under  active  development  in  many  areas  of  computing 
and  information  technology.  Also,  the  number  of  developers  and  users  has  increased. 
But  most  important,  the  benefits  of  systematic  development  and  standardization  of  CT 
have  been  recognized. 

CT  for  STEP  is  intended  to  determine  the  degree  to  which  a developer  has  correctly 
implemented  a standard  AP.  CT  determines  whether  or  not  an  implementation  operates 
correctly  under  normal  operation  for  valid  conditions.  While  CT  evaluates  a single 
implementation  against  a reference  system,  interoperability  testing  uses  two  real 
systems  to  see  if  they  interoperate.  Since  candidate  systems  test  against  a single 
reference  system  (or  exact  copies  thereof),  CT  is  a parallel  process;  whereas 
interoperability  testing  requires  each  system  to  interoperate  with  all  other  systems, 
thereby  having  a sequentisd  nature.  Doing  interoperability  testing  alone  quickly  loses 
its  attractiveness  as  the  number  of  systems  increases.  In  general,  the  cost  of 
interoperability  testing  grows  exponentially.  On  the  other  hand  CT,  while  starting  with 
a higher  fixed  cost  (because  of  the  need  to  build  the  reference  system),  grows 
approximately  in  a linear  fashion  with  N.  This  relationship  is  shown  in  Figure  1. 
Interoperability  testing  is  not  without  its  benefits,  however.  Often,  interoperability 
testing  uncovers  problems  that  went  undetected  during  CT. 

It  should  be  mentioned  that  neither  conformance  nor  interoperability  testing  is  transitive 
in  nature;  i.e.,  if  systems  A and  B pass  CT  with  reference  system  R,  there  is  no 
guarantee  that  they  will  interoperate.  Similarly,  if  A interoperates  with  B,  and  B 
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interoperates  with  C,  there  is  no  guarantee  that  A and  C will  interoperate.  However, 
CT  puts  individual  systems  through  the  same  transitions,  helping  to  take  care  of  non- 
conforming  problems  before  the  vendor  participates  in  interoperability  testing. 
Therefore,  doing  thorough  CT  before  attempting  interoperability  testing  makes  sense. 


A positive  aspect  of 
CT  is  that  once  it  is 
done  on  an 
implementation,  it 
need  not  be  repeated 
until  the 
implementation  or 
abstract  test  suite  is 
modified.  The  intent 
is  to  exercise  as  much 
of  the  logic  inherent  in 
the  implementation  as 
is  practical.  The 
implementation ’s 
responses  to  both 
correct  and  incorrect 
application  protocol 
sequences  are  evaluated.  Due  to  the  complexity  of  the  application  protocols,  it  may  not 
be  realistic  to  try  to  be  exhaustive  during  CT  but  to  use  a representative  sample  of 
operational  sequences.  There  needs  to  be  a strong  agreement  among  standards  bodies, 
users,  and  developers  as  to  what  this  sample  for  a particular  application  protocol  should 
be.  These  operational  requirements  should  be  specified  in  the  standardized  abstract  test 
suites  (ATSs). 

An  early  start  in  STEP  CT  activity  will  likely  lead  to  economy  and  timely  realization 
of  the  benefits  of  product  data  standardization.  This  early  start  should  be  applied  to 
writing  standardized  CT  procedures,  developing  conformance  requirements  and  test 
purposes,  and  producing  supporting  ATSs  for  APs.  Interoperability  testing  will  have 
to  be  done  later,  when  independently  developed  systems  start  interacting.  With  strong 
CT  activity  already  in  progress,  interoperability  testing  will  be  a task  of  lesser 
complexity  and  smaller  magnitude;  a lesson  learned  from  the  IGES  user  community. 
If  no  conformance  testing  is  available,  IGES  users  are  forced  to  depend  solely  on 
interoperability  testing  each  and  every  time  they  acquire  a new  system.  With  no  frame 
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of  reference  for  determining  how  well  a product  conforms  to  a standard,  the  user  is 
dependent  on  trial-and-error  to  obtain  a level  of  confidence. 


2d, 4 The  Cost  of  Conformance  Testing  Versus  the  Cost  of  Not 

Doing  Conformance  Testing 

There  are  a number  of  costs  associated  with  CT: 

• Building  the  organizational  infrastructure  for  doing  CT.  This  involves 
acquiring  a site,  equipping  it,  and  maintaining  administrative  support  for 
abstract  and  executable  test  suite  development  and  later  CT. 

• Developing  CT  expertise.  This  involves  recruiting  appropriately  qualified 
personnel  and  training  them  in  the  related  CT  technology. 

• Building  CT  tools,  ATSs,  and  the  ETSs.  This  is  the  most  important  and 
technology-intensive  part.  This  component  will  take  up  the  largest  of  the 
fixed  cost  and,  if  not  controlled,  may  defeat  the  very  purpose  of  CT  by 
becoming  very  expensive.  In  the  context  of  STEP,  it  assumes  even  more 
importance;  with  several  APs,  an  economical  way  to  generate  high  quality 
AP-specific  ATSs  and  ETSs  is  needed.  The  way  to  achieve  this  is  to  build 
tools  that  will  automate  the  process  of  building  the  ATSs  and  ETSs  as  much 
as  possible.  Since  work  is  already  underway  to  validate  the  APs  prior  to 
their  standardization,  tools  are  being  built  to  facilitate  this  process.  These 
tools  should  be  examined  for  their  potential  use  when  building  the  CT 
system. 

• Maintaining  liaison  with  the  rest  of  the  world.  This  includes  participating 
in  standards  development  meetings  (e.g.,  ISO  TC184/SC4)  and  technology 
development  bodies  (e.g.  those  doing  AP  validation),  and  keeping  in  touch 
with  the  user  and  vendor  communities.  The  CT  community  needs  to  be 
actively  represented  in  standards  bodies  and  implementation  agreement 
bodies,  as  much  to  make  its  task  simpler  as  to  provide  valuable  input  to  the 
standard  and  technology  development  process.  This  participation  should 
continue  not  just  when  the  ATS  and  ETS  are  being  developed,  but  during 
CT  development  and  enhancement.  Maintaining  contact  with  the  users  and 
vendors  is  needed  to  promote  CT,  its  value,  and  its  availability.  Constant 
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contact  with  users  and  vendors  builds  confidence  in  the  CT  process.  Also, 
the  CT  community  can  be  a valuable  resource  to  users  and  vendors. 

• Maintaining  and  offering  the  CT  service.  The  tools,  ATSs,  and  ETSs  have 
to  be  maintained,  upgraded,  and  enhanced  from  time  to  time  to  reflect  the 
revisions  and  additions  to  the  standards.  The  CT  service  itself  involves 
using  acceptable  testing  facilities  and  maintaining  a CT-proficient  staff. 

• Transferring  CT  technology.  Here  we  assume  the  model  of  one  initial  CT 
development  center,  which  will  later  transfer  the  technology  to  multiple 
testing  laboratories  (see  Section  4.3).  In  the  case  of  STEP,  multiple  testing 
sites  will  most  likely  be  necessary  given  the  anticipated  large  number  of 
users  and  vendors. 

The  items  identified  above  all  contribute  to  the  fixed-cost  component  (see  Figure  1)  of 
CT.  The  first  three  however,  are  the  major  contributors.  Careful  design  and 
development  of  CT  tools  are  an  important  means  of  reducing  and  controlling  fixed 
costs.  Also  important  is  structuring  the  APs  to  reduce  duplication  of  ATSs  and  the 
need  to  retest  the  common  aspects  of  APs.  This  is  why  active  representation  in  the 
standards  bodies  is  required.  It  is  possible  to  come  up  with  some  rules  of  thumb,  based 
on  past  experience,  for  estimating  the  above  costs.  While  a laudable  goal,  estimating 
CT  costs  is  beyond  the  scope  of  this  report. 

The  cost  of  not  doing  CT  is  directly  associated  with  the  consequences  of  being  without 
it.  First,  the  earlier  approach  of  doing  ad  hoc  interoperability  testing  v^ll  have  to  be 
followed  in  order  to  have  the  standard  implemented  and  used.  Alternatively,  there  is 
the  distinct  possibility  of  not  having  a standard  adopted  at  all.  For  instance,  the  lack 
of  any  CT  for  IGES,  from  its  inception,  has  been  a contributing  factor  in  making  its 
use  frustrating,  its  adoption  slow,  and  its  cost/benefit  ratio  reduced.  Due  to  the  scope 
of  STEP  and  the  number  of  APs  anticipated,  STEP  CT  will  be  a large  venture. 
Conformance  testing  is  likely  to  be  a key  issue  in  the  success  of  STEP.  The  long-term 
costs  of  not  doing  CT  in  the  United  States  would  probably  retard  the  adoption  of  STEP 
in  the  United  States.  This  could  directly  hurt  the  U.S.  economy  and  have  a significant 
negative  impact  on  U.S.  international  trade  as  well.  The  European  Community  and 
Japan  have  made  a commitment  to  standards  in  general,  and  to  STEP  in  particular. 
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2,1,5  Standards  for  Libraries  of  Data 


Conformance  testing  evaluates  implementations  for  conformance  to  standards.  In  the 
case  of  STEP,  CT  will  generally  use  databases  that  contain  descriptions  of  parts  and 
components.  These  data  repositories  or  libraries  must  themselves  be  standards- 
conforming. For  example,  the  geometric  information  contained  in  a STEP  data  file 
should  be  accessible  by  a STEP-conforming  CAD  system.  Apart  from  the  development 
of  data  libraries  to  help  in  conducting  CT,  hardware  vendors  are  likely  to  build  data 
libraries  as  well.  STEP  data  files  must  be  readable  by  the  implementation  under  CT. 
In  addition  to  standards  for  STEP  data  files,  there  should  be  standards  for  libraries  of 
such  STEP  data  files. 


2,1,6  Conformance  Levels 

Past  experience  has  demonstrated  that  allowing  partial  conformance  will  only  lead  to 
chaos  and  confusion  in  the  minds  of  users.  Partially  conforming  applications  may  be 
less  likely  to  interoperate,  thereby  undermining  the  most  fundamental  purpose  of  CT. 
Moreover,  they  encourage  misrepresentation  and  misuse.  Finally,  attempts  to  grant 
partial  conformance  will  render  the  task  of  developing  CT  systems  and  doing  CT  much 
more  complex. 

Recommendation 

• Partial  conformance  must  be  treated  carefully;  full  conformance  must  be  the 
goal 

Generally,  products  would  either  be  conforming  or  nonconforming.  In  other 
standards  there  have  been  multiple  specification  classes  to  which  applications  may 
choose  to  conform.  In  the  STEP  standard,  current  strategy  seems  to  be  to  enforce 
total  conformance  to  a given  AP.  The  logic  behind  this  is  that  a well-defined, 
stand-alone  part  of  an  AP  should  become  a new  AP.  Provisional  conformance 
could  be  considered  if  time  limits  (e.g.,  one  year)  for  correcting  the  non- 
conformities were  enforced.  Allowing  provisional  conformance  permits  greater 
competition,  results  in  more  products  in  the  market,  and  lessens  the  "up-front 
costs"  burden  to  the  implementor.  Ultimately,  within  an  AP,  full  conformance 
should  be  required. 
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2,1,7  Dependencies  Between  Application  Protocols 

The  primary  objective  of  STEP  is  to  provide  for  meaningful  communication  of  product 
data  from  one  industrial  application  to  another.  In  STEP,  APs  will  be  interrelated.  The 
scopes  of  proposed  APs  should  be  reviewed  carefully  to  ensure  no  more  or  less  is 
supplied  than  the  justified  industrial  need  stated  on  the  AP  project  approval  summary 
sheet.  This  raises  the  issue  of  testing  a product  for  conformance  to  referenced  APs  to 
the  extent  the  product’s  conforming  behavior  may  be  affected  by  these  references.  The 
question  of  how  far  to  test  and  when  to  stop  is  not  directly  addressed  by  STEP.  In 
ITI’s  opinion,  the  standards  development  activities  in  STEP  have  not  progressed  far 
enough  to  help  provide  answers  to  this  question. 

Recommendations 

• Further  study  is  needed  to  determine  how  APs  are  to  interoperate  and  what 
the  implications  on  CT  are. 

• The  ISO  TC184ISC4  working  groups  on  CT  and  AP  methodologies  need  to 
address  in  the  proposed  technical  report  for  AP  Development  Guidelines,  the 
issue  of  AP  interoperability  and  how  it  relates  to  CT  . 


2,2  Strategic  Issues 

An  objective  of  the  DoD’s  CALS  (Computer-aided  Acquisition  and  Logistic  Support) 
initiative  is  to  encourage  the  adoption  and  use  of  STEP  in  DoD  procurements.  To 
make  this  a reality,  developers  and  users  will  have  to  have  confidence  in  STEP  and 
work  to  build  skills  in  using  it.  A key  objective  is  to  protect  the  interests  of  the 
various  participants  involved  in  the  adoption  of  STEP.  These  participants  are  the 
developers,  the  users,  the  standards  bodies,  and  the  testing  agents.  The  following 
subsections  discuss  in  detail  the  various  issues  that  confront  these  four  main 
participants  in  relation  to  CT  in  general,  and  STEP  CT  in  particular. 


2,2,1  U,S,  Involvement  in  STEP  Conformance  Testing 

There  are  several  reasons  for  the  United  States  to  be  involved  in  STEP  CT. 
Encouraging  CT  activity  is  a way  to  build  indigenous  application  protocol  expertise. 
This  will  prove  invaluable  later  in  the  development  of  conforming  implementations  and 
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enable  U.S.  vendors  to  acquire  and  maintain  a competitive  advantage  and  encourage 
users  to  adopt  STEP-based  products. 

Since  CT  inevitably  raises  issues  about  the  standard  itself,  it  becomes  an  important 
instrument  in  the  evolution  of  the  standard.  The  United  States  needs  to  play  an  active 
role  in  CT  and  the  evolution  of  STEP. 

A significant  U.S.  effort  in  development  and  testing  activities  would  make  clear  the 
U.S.  government’s  intentions  and  reinforce  earlier  policy  statements  in  support  of  the 
standard.  This,  in  turn,  would  provide  the  impetus  needed  for  corporate  (both  user  and 
developer)  buy-in.  The  anticipation  of  future  business  needs  and  the  need  to  be 
prepared  for  them  would  thus  trigger  a development  of  STEP  expertise.  Another 
important  benefit  of  indigenous  CT  activity  is  that  the  organizations  involved  in  it 
become  a resource  focused  on  making  STEP  a reality.  Vendors  and  developers  can  use 
this  resource  for  help  in  standards  implementation  and  sorting  out  interoperability  and 
acceptance  issues. 

Finally,  developing  early  CT  capability  for  an  AP  brings  with  it  the  opportunity  of 
early  understanding  of  the  standard.  For  STEP,  the  ability  of  U.S.  vendors  and  users 
to  leverage  this  opportunity  to  best  advantage  could  have  significant  economic 
implications.  Conversely,  if  left  to  others,  this  advantage  is  lost,  and  a dependence  on 
non-U.S.  testing  bodies  would  result. 

U.S.  involvement  in  STEP  CT  will  benefit  the  entire  development  cycle  fi'om  the 
development  of  early  conforming  implementations  to  the  creation  and  nurturing  of  a 
vigorous  and  competitive  market  for  products.  Ultimately,  this  will  lead  to  faster 
adoption  of  STEP  by  U.S.  industry  through  the  rapid  appearance  of  more  and  better 
STEP-conforming  products. 

Recommendation 

• It  is  important  that  a strong  participation  in  CT  be  part  of  US.  STEP 
activities. 


222  CT  System  Openness 

The  idea  that  CT  systems  should  be  open  has  considerable  merit.  Openness  of  the  CT 
system  means  CT  ATSs,  ETSs,  associated  test  tools,  and  associated  documentation  are 
readily  available  to  all  interested  parties.  Developers  and  users  could  use  the  CT 
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system,  voice  their  concerns  about  problems  and  inconsistencies,  report  bugs,  and 
thereby  provide  feedback  for  improvement. 

Early  availability  of  the  CT  system  will  encourage  early  development  of  STEP- 
conforming  products.  This  will  improve  product  quality  and  reduce  developer  costs. 
Familiarity  with  the  CT  system  and  use  during  the  development  process  will  give 
developers  a sense  of  confidence  in  and  partnership  with  the  CT  process. 

Concern  has  been  voiced  that  making  the  CT  systems  completely  open  might  tempt 
developers  to  devote  more  of  their  energies  to  passing  the  CT  and  less  to  building  a 
better  product.  This  concern  is  not  supported  by  experience  in  other  domains. 
Complete  CT  systems  are  now  available  for  many  networking  protocols,  and 
commercial  organizations  are  building  their  own  testing  laboratories  for  internal  testing 
and  quality  assurance.  Competitive  pressures  force  vendors  to  produce  good  products. 

An  easily  accessible  CT  system  will  expedite  disseminating  and  popularizing  the 
standard.  It  will  also  help  users  determine  their  own  requirements. 

Recommendations 

• Make  CT  systems  available  for  use  to  interested  developers  at  nominal  cost. 

• Create  a forum  for  interested  parties  to  contribute  toward  a better  CT 
program.  Inputs  for  improving  the  standards,  the  CT  system,  and 
implementation  methods  could  be  gathered  through  such  a forum. 


2,2.3  DoD  Security  Concerns 

Defense  organizations  and  contractors  will  develop  STEP-conforming  software 
apphcations  as  a result  of  the  CALS  initiative.  Often  such  software  will  work  in  a 
secure  environment  or  access  secure  or  classified  resources  Gike  databases).  In  most 
cases  the  software  applications  can  be  isolated  fi'om  the  classified  data.  This  may  not 
be  possible  for  some  classes  of  systems,  e.g.  embedded  systems.  Three  methods 
performing  CT  on  such  applications  exist:  (1)  installing  identical  systems  in  the  testing 
laboratory  for  the  duration  of  the  conformance  assessment  process;  (2)  "taking"  the 
testing  laboratory  to  the  application;  or  (3)  allowing  vendor  "self-testing"  as  declaration 
of  conformity.  In  each  of  these  situations  the  security  of  such  applications  could  be 
compromised.  Performing  CT  on  sensitive  applications  and  implementations,  given 
their  nature  and  the  access  restrictions  they  carry,  is  a major  concern. 
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Recommendations 


• Two  alternative  recommendations  to  this  problem  are  suggested: 

DoD  develops  its  own  CT  laboratory  and  does  all  classified  testing  in 
it. 

DoD  secures  a security-cleared  contractor  who  acquires  CT  capability 
and  provides  the  service  to  DoD  in  a secure  environment. 

It  is  difficult  to  select  the  best  choice;  the  first  approach  may  be  suitable  if  there 
is  sufficient  software  to  be  tested  but  may  take  some  time  to  implement.  A 
contracting  arrangement  might  prove  to  be  more  economical,  especially  if  the 
volume  of  CT  does  not  warrant  a dedicated,  in-house  lab. 


22,4  DoD  Adoption  Concerns 

The  Adoption  Issue:  DoD  is  probably  the  largest,  single  stakeholder  in  STEP.  STEP 
is  also  an  important  part  of  CALS.  DoD  wants  to  see  suppliers  adopt  STEP  so  that 
consistent  means  of  product  information  exchange  can  be  established.  Therefore,  a 
favorable  perception  of  STEP  by  DoD  contractors  is  needed  to  ensure  adoption.  The 
entire  CT  effort  is  aimed  at  achieving  this  objective  by  expediting  the  adoption  process 
and  making  it  happen  more  efficiently  and  cost-effectively. 

Recommendation 

• DoD  should  set  realistic  adoption  milestones  and  actively  support  their 
achievement. 

DoD,  as  a huge  buyer  of  information  technology  goods  and  services,  wields 
enormous  influence  with  its  suppliers.  DoD  could  use  this  influence  to  drive 
vendor  agreement  to  build  STEP  application  protocols;  insist  on  conformance 
testing  of  application  systems  which  implement  them;  and  use  them  to  meet  DoD 
procurement  requirements. 
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2.2.5  Vendor  Community  Needs 

Fulfilling  the  needs  of  vendors  and  developers  would  go  a long  way  toward 
encouraging  widespread  adoption  of  standards  and  education  about  CT.  To  many,  CT 
represents  a relatively  new  approach  to  working  with  standards.  It  is  not  really  new. 
Underwriters  Laboratory  has  been  doing  this  kind  of  work  for  years  on  hardware 
implementations.  The  Federal  Government  has  been  doing  CT  of  computer 
programming  language  software  implementations  for  years.  Even  Consumers  Union 
does  CT  on  a targeted,  ad  hoc  basis.  It  is  important  to  promote  CT  and  inform  the 
broad  STEP  community  of  its  benefits.  This  would  help  vendors  understand  the  need 
for  conformance-tested  products  and  the  competitive  advantage  that  accrues  from  CT. 
This  promotional  task  has  to  be  a continuous  process. 

As  vendors  start  developing  STEP  conforming  products,  there  will  be  a need  for 
identified  personnel  that  will  answer  standard-related  questions,  help  in  application 
protocol  implementation  and  CT  issues,  and  provide  a forum  for  discussion  and 
problem  solving. 

Recommendations 

• A forum  to  discuss  CT  issues  should  be  created.  Standards  bodies  typically 
provide  such  a facility  in  some  form  or  another  (annual  or  semi-annual 
conferences,  publications,  newsletters,  bulletin  boards,  etc.),  but  an 
additional  forum  for  CT  is  necessary. 

Outputs  fi'om  such  a forum  would  include:  regularly  published  news  and 
information,  workshops,  education  for  vendors  and  developers,  and  promotion  of 
CT.  This  forum  would  provide  an  opportunity  to  disseminate  information  and 
allow  developers  to  report  problems,  request  information,  and  seek  related 
assistance  to  perform  better  in-house  testing. 

Seminars  and  workshops  to  do  training  on  CT  tools  and  methods  should  be 
organized  through  this  same  forum. 

• For  vendors  to  use  the  open  CT  systems  as  recommended  above,  training  to 
use  the  CT  tools  and  methods  is  necessary. 
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22,6  Accreditation 


Here  we  assume  the  following  model:  Initially  there  would  be  one  team  for  developing 
the  CT  system.  Since  a lot  will  depend  on  this,  the  long-term  effects  of  choosing  a 
particular  team  should  be  kept  in  mind.  Once  the  CT  system  has  been  developed  to 
a level  where  CT  can  begin,  multiple  testing  laboratories  would  be  set  up  and 
accredited. 

Recommendation 

• Since  one  of  NIST  s primary  functions  is  to  promote  standards  for 
government  use,  it  should  be  considered  as  a candidate  to  accept  the 
responsibility  of  directing  and  performing  the  initial  CT  development. 

• The  US  should  encourage  and  participate  in  international  collaborative 
development  of  STEP  conformance  testing  procedures  and  work  toward  the 
mutual  recognition  of  conformance  testing  services. 

Therefore,  while  the  funding  agencies  and  NIST  should  work  together  to  come 
up  with  a consensus  approach  for  accrediting  testing  laboratories,  detailed 
decisions  could  be  left  to  NIST  as  an  impartial  agent. 

World-wide  development  of  STEP  CT  does  have  United  States  representation,  but  in 
limited  strength  relative  to  the  rest  of  the  world.  It  is  important  for  the  United  States 
to  coordinate  its  position  in  forums  such  as  ISO.  The  United  States  should  encourage 
harmonization  activities  and  therefore  must  organize  itself  into  a position  of  unity, 
confidence,  and  strength  with  the  rest  of  the  world.  Government  participation, 
leadership,  and  teaming  with  industry,  as  is  common  within  Europe  and  Japan,  would 
go  a long  way  in  this  area. 


2.2.7  Assigning  Conformance  Testing  Responsibilities  to  Testing 
Laboratories 

Let  us  assume  staff  of  the  National  PDES  Testbed  (NPT),  working  with  others,  will 
initially  develop  the  CT  system.  They  would  then  distribute  the  CT  system  and  transfer 
the  technology  to  organizations  that  want  to  be  in  the  business  of  being  third-party  or 
in-house  vendor  test  centers,  once  accredited,  such  organizations  would  be  designated 
as  testing  laboratories. 
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For  widespread  adoption  of  a standard,  there  has  to  be  a commitment  shared  by 
interested  parties  to  make  the  standard  work.  A testing  laboratory  would  have  to 
shoulder  its  share  of  this  commitment.  The  most  fundamental  role  of  the  testing 
laboratory  is  to  see  that  implementations  work. 

From  the  standard  development  point  of  view,  a testing  laboratory  should  also  play  a 
participatory  role  by  providing  feedback  to  the  standard  development  process.  Such 
input  should  reflect  CT  issues  and  closely  address  the  problems  of  product  developers. 

Another  level  of  responsibility  of  testing  laboratories,  though  indirect,  may  be  to  ensure 
that  STEP,  as  a standard,  is  efficiently  and  economically  implemented.  Failure  to 
satisfy  such  efficiency  and  economy  criteria  often  impacts  how  the  standard  itself  is 
perceived.  This  aspect  is  beyond  the  scope  of  CT,  but  particularly  helpful  in  promoting 
STEP’S  usefulness.  Efficiency  and  economy  issues  of  implementations  are  often  dealt 
with  in  workshops  between  testing  laboratory  personnel  and  vendors. 

Recommendation 

• Ensure  applicant  testing  laboratories  are  both  committed  and  skilled  in 
conformance  testing  and  the  technical  aspects  of  STEP. 


22,8  Abstract  Test  Suites  (ATSs) 

Leaving  unassigned  the  responsibility  of  developing  ATSs  would  result  in  different 
organizations  (developers,  testing  laboratories,  ad-hoc  industry  groups  or  consortia) 
building  their  own  and  claiming  conformance  to  them.  This  would  lead  to 
inconsistencies  in  products  and  CT,  cause  confusion,  and  ultimately  slow  down  the 
adoption  process. 

Recommendations 

• Standards  bodies  should  be  responsible  for  the  ATSs. 

• ATSs  should  be  normative  parts  of  STEP. 

• The  ATS  should  be  a normative  requirement  for  a given  AP. 

The  actual  development  of  ATSs  could  be  done  by  anyone  (testing  laboratories, 
consortia,  etc.)  as  long  as  they  are  reviewed  by  the  standards  committees  and 
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accepted  as  part  of  the  standards.  Developers  of  such  ATSs  should  be  committed 
to  keeping  the  ATSs  in  the  public  domain  and  internationally  approved  as  part  of 
STEP. 


22.9  Executable  Test  Suites  (ETSs) 

It  is  possible  to  build  multiple  ETSs  from  the  same  ATS.  The  need  for  this  may  arise 
due  to  various  platform  and  implementation  choices  made  by  product  developers. 

Recommendation 

• Despite  the  non-unique  nature  of  an  ETS,  the  development  of  the  first  ETS 
for  an  ATS  should  be  done  by  those  building  the  ATS,  or  in  close 
cooperation  with  them. 

This  would  help  to  identify  and  correct  any  problems  with  future  versions  of  the 
standard  ATS. 


2.3  Development  Issues 

A number  of  issues  have  to  be  addressed  before  a strong  CT  infrastructure  is  built  and 
brought  into  operation.  The  main  resource  needs  of  CT  are  those  of  expertise,  tools, 
a lead  organization,  and  funds  to  support  such  activities.  Building  STEP  CT,  whether 
in  the  United  States  or  elsewhere,  will  require  people  with  the  appropriate  technical 
expertise.  The  first  major  task  of  this  CT  system  development  team  would  be  to 
specify,  design,  build,  and  test  the  CT  system  and  associated  documentation.  There 
will  be  a need  for  an  organizational  infrastructure  to  support  these  activities.  All  these 
tasks  also  require  funding. 

There  needs  to  be  a technical  liaison  between  the  CT  system  development  team  and 
ISO  TC184/SC4.  Contact  has  to  be  maintained  through  active  participation  in  SC4 
Working  Groups.  Contact  also  has  to  be  maintained  with  organizations  and  industry 
groups  developing  ATSs.  To  keep  abreast  with  vendor  activities  and  issues  of 
interpretation,  CT  system  development  staff  should  attend  implementors’  agreement 
meetings,  trade  shows,  and  symposia. 
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Another  set  of  tasks  includes  continually  assessing  and  controlling  the  quality  of  the 
CT  systems  being  built  by  the  CT  development  team.  Only  the  initial  stages  of  CT 
system  and  tool  development  will  be  handled  by  the  CT  development  team  as  a single 
entity.  Once  the  early  versions  of  the  transferable  CT  system  have  been  developed, 
many  organizations  would  have  the  opportunity  to  become  testing  laboratories.  These 
organizations  should  be  accredited  and  a standard  procedure  should  be  established  to 
do  this. 


2,3  J Abstract  Test  Suite  (ATS)  Development 

It  has  been  discussed  earlier  that  standards  bodies  should  include  developing  the  ATSs 
as  part  of  the  standards-making  process.  ATSs  could  be  developed  by  any  independent 
organization,  but  responsibility  for  their  ratification  and  acceptance  belongs  to  ISO 
TC184/SC4. 

Recommendations 

• When  the  CT  development  activity  begins,  some  initial  start-up  funding  may 
be  needed  to  pay  for  the  development  of  ATSs  for  those  APs  released 
without  an  ATS. 

• Once  the  CT  activity  stabilizes,  the  cost  of  developing  ATSs  for  new  (and 
changing)  APs  should  be  borne  by  the  users  of  the  APs.  For  example,  ATSs 
for  APs  supporting  the  automotive  industry  should  be  paid  for  by  the  auto 
industry. 

One  approach  to  ATS  development  is  that  one  center  would  develop  the  CT  systems 
and  tools  for  a small,  self-contained  piece  of  STEP  (like  a group  of  related  APs).  This 
would  give  the  primary  sponsor  of  the  activity  opportunity  for  input  and  assure 
interested  parties  the  task  is  not  getting  bogged  down  in  the  complexities  of  very  large 
system  development.  Starting  small  would  make  it  possible  to  monitor  the 
development  activity,  keep  quality  high,  and  keep  costs  down.  As  the  task  progresses, 
the  scope  could  increase.  This  approach  towards  incremental  development  fits  well 
with  all  the  strategic  issues  discussed  in  Section  2.2. 
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2,3,2  Funding  Conformance  Testing 

Conformance  testing  start-up  costs  have  generally  been  borne  by  interested  industry 
groups  or  consortia  and  applications  developers.  Of  course,  any  costs  faced  by 
developers  eventually  filter  down  to  the  user  in  the  form  of  higher  prices.  This  effect 
would  be  more  significant  if  developers  are  expected  to  pay  for  the  start-up  as  well, 
since  the  development  of  the  CT  activity  would  require  a significant  initial  investment. 
Ultimately,  the  benefits  of  CT  to  users  of  STEP  implementations  have  to  be  seen  as 
greater  than  the  cost.  If  users  fund  the  start-up  of  a CT  service,  the  developer  may 
incur  a lower  fee  for  such  a service.  Low  fees  will  attract  developers  and  help  get  the 
CT  activity  off  the  ground.  Once  stabilized,  the  rise  in  revenue  from  CT  should  make 
it  possible  to  reduce  the  subsidy.  In  the  long  run  CT  should  be  a vendor-supported 
activity.  Therefore,  if  CT  costs  are  reasonable,  it  encourages  more  vendors  to  build 
affordable  implementations  while  still  undergoing  CT. 

There  are  other  issues  to  consider: 

• Does  the  complexity  of  the  application  protocol  and  magnitude  of  the 
ETS  affect  fees? 

• Do  all  vendors  pay  the  same  or  do  smaller  companies  pay  less  than 
larger  ones. 

Recommendation 

• When  conformance  testing  first  begins,  vendors  should  not  be  expected  to 
pay  more  than  nominal  sums  to  have  their  products  tested. 


2,3,3  Legacy  Issues 

Retesting  may  be  needed  when  developers  claim  conformance  with  newer  versions  of 
standards,  or  modify  their  products  to  the  existing  standards.  Thus,  a changing 
standard  or  product  drives  the  need  for  retesting.  There  are  two  issues  to  consider: 
availability  and  cost.  If  the  CT  systems  are  easily  available  (Section  2.2.2),  vendors 
could  do  some  sort  of  strictly  controlled  provisional  conformance  testing  in-house,  if 
their  original  base  system  was  already  deemed  conforming.  This  would  drive  costs 
down.  Also,  the  CT  assessment  process  costs  should  be  reasonable  to  encourage 
vendors  to  get  CT  done  for  upgraded  products  early.  Moreover,  if  CT  is  carefully 
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implemented  in  a modular  manner,  retesting  may  only  involve  CT  of  those  portions 
that  have  undergone  changes. 

Recommendation 

• As  mentioned  earlier,  if  CT  is  done  right  (open,  modular,  reasonably 
priced),  retesting  should  not  be  an  insurmountable  problem.  Also,  as  APs 
stabilize,  retesting  would  cease  to  be  an  important  issue. 

23 A Tools 

One  point  this  report  has  emphasized  is  the  need  for  automating  the  conformance 
assessment  process  and  the  ETS  development  process.  The  way  to  achieve  this  is 
through  software  tools.  CT  tools  reduce  dependence  on  the  expertise  of  the  CT  system 
operator  and  also  reduce  the  chances  of  CT  system  operator  error.  CT  development 
tools  help  build  new  CT  system  components,  thus  reducing  costs. 

The  following  is  a list  of  the  tools  potentially  needed  for  conformance  testing.  They 
can  be  classified  into  three  types:  CT  System  Tools,  Client  Support  Tools,  and  CT 
Service  Tools.  Some  of  the  tools  listed  below  will  become  a necessary  part  of  any  CT 
system  for  virtually  any  application  protocol.  Others  are  less  common,  but  valuable, 
nonetheless. 


23A.1  CT  System  Tools 

Test  Engines  - These  are  the  generally  invariant  core  drivers  of  test  execution  that 
probe  and  detect  the  behavior  of  the  System  Under  Test  (SUT).  They  perform  common 
activities  required  by  the  abstract  and  executable  test  cases  (ATCs  and  ETCs). 

Executable  Test  Cases  - These  are  the  data  sets  or  instructions  that  when  coupled  with 
a test  engine,  control  a test  engine  into  stimulating  particular  sets  of  behaviors  from  an 
Implementation  Under  Test  (lUT). 

Test  Session  Controllers  - These  tools  sequence  and  orchestrate  the  execution  of  other 
CT  tools,  with  the  objective  to  see  the  procedural  requirements  of  conformance  test 
sessions  are  properly  carried  out,  or  if  not,  duly  noted. 
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Loggers  - These  tools  record  the  most  basic  activities  of  CT  system  execution  for 
consumption  by  other  tools,  or  for  audit  of  test  execution  events. 

Monitors  - These  tools  observe  and  report  on  SUT  behavior. 

Analyzers  - These  are  tools  which  substitute  for  both  tedious  clerical  editing  and 
insightful  expert  interpretation. 

Report  Generators  - These  tools  collect  the  important  outputs  of  the  other  CT  system 
tools  into  a report  format  suitable  for  external  distribution. 

Preparation  Tools  - These  tools  collect  information  about  an  SUT,  and  record  it  for 
use  by  other  CT  system  components. 

Data  Management  Tools  - These  tools  enable  organized  archival  storage  and 
examination  of  test  data. 

Debuggers  - These  tools  are  a required  by-product  of  CT  system  development  in  that 
they  can  be  used  to  evaluate  and  possibly  correct  CT  system  operation. 

Build  Tools  - These  tools  extract  appropriate  information  about  the  tests  to  be 
performed,  and  configure  and  activate  the  CT  system  accordingly. 

Clean-Up  Tools  - These  tools  de-activate  and  de-configure  the  CT  system,  along  with 
safely  archiving  captured  test  results. 

Training  Tools  - These  tools  can  be  used  to  educate  the  testing  laboratory  staff  on  the 
use  of  the  CT  system. 


23.4,2  Client  Support  Tools 

CT  System  Interfaces  - These  are  the  necessary  hooks  that  a client  must  provide  to  the 
CT  system  in  order  for  the  SUT  to  be  examined. 

Test  Support  Tools  - These  may  be  required  of  a client,  executable  in  the  SUT,  so  that 
the  SUT  is  testable. 
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Test  Responders  - These  are  tools  and  functions  sometimes  required  by  vendors  to  test 
their  implementations.  These  could  be  sample  code  that  is  portable  or  a detailed 
specification. 

Test  Procedures  - These  are  established  to  help  clients  get  their  products  tested  easily. 
These  procedures  should  be  precise,  simple,  and  easy  to  carry  out. 

Report  Generators  - These  are  tools  to  format  the  CT  results  for  easy  perusal  by 
vendors  and  users. 

Training  Tools  - These  tools  can  simplify  the  client’s  efforts  in  the  CT  operation 
activities  by  providing  key  information  about  such  aspects  as  the  CT  process  and 
requirements  of  an  SUT. 


2,3,43  Testing  Laboratory  Service  Tools 

Test  Engine  - As  described  above,  it  is  the  automated  apparatus  which  can  stimulate 
and  observe  an  SUT’s  behavior  in  a controlled  manner. 

System  Diagnostics  - These  tools  can  be  engaged  to  validate  proper  operation  of  the 
CT  system,  the  client  system,  or  the  CT  environment. 

Remote  Testing  - These  tools  can  be  used  to  execute  tests  over  telecommunication 
facilities  if  required  by  special  client  needs. 

Remote  Control  - These  tools  further  enhance  remote  CT  by  providing  clients  with  the 
capability  to  control  the  CT  environment  for  either  pre-conformance  testing  preparation 
or  debugging. 

Scheduling  - These  tools  are  the  classic  software  packages  which  are  quite  valuable  in 
moderate-to-high  activity  operations. 

Logistics  Coordination  - These  tools  help  in  facilitating  coordinated  interactions 
between  CT  service  parties. 

Accounting  - These  tools  assist  in  tracking  the  costs  of  both  client  and  testing 
laboratory  operations. 
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Forms  Management  - These  tools  assist  in  the  generation,  preparation,  and  organization 
of  blank  and  completed  reports,  forms,  and  data  sets. 

Report  Production  - These  tools  help  with  the  collection,  organization,  and  preparation 
of  reports  for  "official"  results  reporting,  CT  plan  generation,  and  other  external 
document  publication. 

Recommendation 

• The  specific  set  of  tools  needed  for  STEP  CT  needs  to  be  determined  as  part 

of  the  CT  system  development. 

A full  set  of  CT  tools  is  probably  too  much  to  develop  as  part  of  the  initial  CT 
work.  As  an  initial  set  of  specific  tools  is  developed,  they  will  help  in  further  CT 
and  product  development.  The  earlier  tools  become  available,  the  quicker  CT  of 
standard-conforming  products  will  occur.  The  lists  presented  above  are  an  initial 
suggestion  of  tools  likely  to  be  useful. 
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5.  Evaluation  Procedures  for  STEP  Conformance  Testing 


The  success  of  STEP-based  application  systems  will  depend  on  the  representation  of 
product  data  in  a form  completely  independent  of  any  particular  computer-aided 
software  system.  This  basic  requirement  brings  with  it  the  need  for  file  exchange 
capabilities,  sharing  of  product  databases,  and  device-independent  archiving  and  de- 
archiving capabilities. 

To  have  confidence  in  the  verdict  of  a CT  system,  the  system  itself  needs  to  be 
evaluated  as  do  the  ATSs  for  the  various  application  protocols  that  will  be  used  to 
generate  the  ETS  run  on  the  CT  system.  Thus,  evaluation  applies  suitably  defined 
criteria  to  candidate  CT  systems  and  offers  authoritative  approval  or  disapproval.  In 
this  section  we  focus  on  the  evaluation  of  CT  systems,  ATSs,  and  ETSs  for  CT  and 
offer  some  recommendations  for  evaluation  procedures. 

One  important  caveat  needs  to  be  raised  in  regard  to  CT  system  approval.  The 
evaluation  process  must  not  fall  into  a gatekeeper  mentality.  The  processes  need  to  be 
designed  to  encourage  CT  and  the  operation  of  testing  laboratories.  If  the  attitude 
becomes  one  of  keeping  testing  laboratories  out  rather  than  helping  them  become 
operational,  then  the  CT  system  will  be  inadequate,  if  not  counterproductive.  Likewise, 
the  testing  laboratories  will  need  to  take  the  approach  of  helping  their  customers  create 
high-quality,  standards-conforming  products. 


3.1  CT  System  and  Service  Evaluation 

Criteria  are  needed  for  the  process  of  evaluation  of  both  CT  systems  and  services.  The 
CT  system  includes  both  the  hardware  and  software  environment  and  the  test  suites 
used  in  the  conformance  assessment  process.  Automation  within  CT  systems 
contributes  to  the  ability  of  testing  services  to  carry  out  the  procedures  and  policy  for 
STEP  conformance  testing.  Both  the  CT  systems  and  services  fi’om  testing  laboratories 
need  to  be  accredited.  The  following  subsections  introduce  some  important  criteria  and 
related  recommendations. 
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3dJ  CT  System  Environment 


The  hardware  and  software  environments  used  in  a CT  system  will  have  to  be 
evaluated.  Since  multiple  CT  environments  could  be  built  to  meet  the  criteria  for 
acceptance,  any  such  CT  system  must  not  give  commercial  advantage  to  any  particular 
cancUdate  product  during  CT. 

Recommendations 

• The  CT  environment  specifications  should  be  easily  available  to  all 
interested  parties  who  would  have  the  opportunity  to  voice  their  concerns, 
if  any,  and  request  reasonable  modifications  in  the  early  stages  of  CT  system 
development. 

• The  acceptance  of  a CT  environment  should  not  favor  some  product  vendors 
over  others. 

• The  CT  environment  must  be  an  "open"  platform  to  and  from  which  the 
porting  of  CT-related  software  will  be  easily  accomplished. 

• The  providers  of  CT  systems  should  be  "open"  to  licensing  by  testing 
laboratories  and  other  users. 

• The  CT  environment  should  be  designed  to  be  portable  for  use  at  client 
sites. 

• The  CT  environment  should  not  impose  architectural  requirements  on  SUTs 
for  their  successful  CT;  i.e.,  the  cause  for  failure  of  a test  or  the  inability  to 
perform  it  should  not  be  architectural  incompatibility  between  the  CT  system 
and  the  SUT. 

3,12  CT  System  Availability 

Product  vendors  need  easy  access  to  CT  systems  to  be  used  as  development  and 
debugging  tools  throughout  the  process  of  product  development.  Thus  CT  systems 
should  be  reasonably  priced. 
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Recommendations 


• The  cost  of  acquiring  (buying,  leasing,  or  sharing)  a CT  system  should  not 
be  such  that  it  would  prohibit  widespread  use. 

• The  cost  of  developing  STEP  conformance  testing  systems  should  be 
distributed  by  seeking  international  collaboration  and  harmonization. 

• CT  systems,  tools,  and  executable  test  suites  built  for  open  computing 
environments  will  enlarge  the  market,  allowing  developers  to  reduce  the  cost 
per  unit . 

Since  the  cost  of  purchasing  some  CT  systems  may  preclude  their  in-house  use 
by  some  vendors,  arrangements  to  make  them  available  at  reduced  costs  or  for 
cost  sharing  should  be  put  in  place. 


3J,3  CT  System  Characteristics 

CT  systems  need  to  possess  some  basic  characteristics  which  assure  their  integrity, 
exhibit  their  independence  of  the  CT  environment,  and  provide  evidence  of  their 
usability: 


• Repeatability  of  CT  results 

• Reporting  and  easy  comparability  of  CT  results 

• Auditabihty  of  CT  results 

• Usabihty  (ease  of  installation  and  use)  of  the  CT  system 

3d, 3d  Repeatability  of  Conformance  Testing  Results 

To  assure  a high  level  of  confidence  in  the  results  of  any  conformance  assessment 
process  and  the  credibility  of  the  testing  laboratory,  CT  results  must  be  repeatable.  A 
combination  of  the  same  ETS,  CT  system,  and  the  HIT  must  produce  identical  results 
over  multiple  runs. 
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Recommendations 


• A candidate  CT  system  should  be  subjected  to  a calibration  process  to 
evaluate  its  repeatability.  This  evaluation  should  include  the  CT  system’s 
responses  to  both  error-free  and  erroneous  implementations  of  STEP 
products. 

To  maintain  a high  degree  of  confidence  in  the  performance  of  this  calibration, 
the  implementation(s)  to  be  used  should  be  selected  from  a known,  fairly  large 
list.  The  results  of  this  exercise  will  form  the  basis  for  the  judgement  of  whether 
repeatable  results  can  be  expected. 

• Because  repeatability  of  test  results  may  be  affected  by  CT  system  operator 
error  and  result  generation  error,  a high  degree  of  automation  is 
recommended. 

3.13.2  Reporting  and  Easy  Comparability  of  Conformance  Testing 
Results 

A CT  system  must  report  information  that  will  allow  the  production  of  a Conformance 
Test  Report  (CTR)  for  each  implementation  tested  against  each  relevant  AP.  The  CTR 
shall  specify  for  every  test  run  one  of  three  outcomes:  pass,  fail,  or  inconclusive.  No 
other  verdicts  are  allowed.  The  CT  system  should  thus  allow  logging  of  each  test 
outcome  during  the  testing  process. 

Moreover,  the  CTR  generated  should  be  independent  of  the  CT  environment.  Thus  for 
the  same  SUT,  the  CTR  generated  at  different  test  sites  using  the  same  abstract  test 
method,  should  be  comparable. 

Recommendations 

• The  CT  system  should  possess  the  capability  of  using  its  logs  to 
automatically  generate  a CTR. 

• Formats  and  procedures  as  defined  in  ISO  10303-32^  for  generating  CTRs 
should  be  adhered  to. 


^ISO  10303-32,  Working  draft  in  TC184/SC4/WG6,  "Conformance  testing  methodology  and 
framework:  requirements  on  testing  laboratories  and  clients." 
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Procedures  in  ISO  10303-34^  for  executing  test  cases  should  be  followed. 


These  procedures  should  be  built  with  flexibility  in  mind  and  should  consider 
situations  in  which  repetitions  of  specific  test  case  executions  may  be  warranted. 


3,1,33  Auditability  of  Conformance  Testing  Results 

A CT  process  usually  includes  an  arbitration  subprocess.  To  effectively  arbitrate  a 
dispute  between  product  vendor  and  testing  laboratory,  detailed  information  on  the 
inputs  and  outputs  for  the  execution  of  a test  suite  is  required. 

Recommendation 

• The  ability  to  document  the  actual  procedure  of  a test  campaign  and  log  its 
details  should  be  built  into  the  CT  system.  These  logs  should  then  be  used 
as  the  basis  of  the  arbitration  process. 

3, 1,3, 4 Usability  of  the  Conformance  Testing  System 

The  CT  system  builder  should  not  assume  that  the  CT  system  operator  will  be 
STEP-knowledgeable  or  conversant  with  CT  technology.  It  should  be  possible  to 
quickly  train  a lay  computer  user  to  be  a CT  system  operator.  Also,  the  purposes  for 
building  and  using  a CT  system  is  defeated  if  many  parts  of  the  conformance 
assessment  process  are  left  non-automated.  The  CT  system  should  be  straightforward 
to  install,  configure,  and  operate.  CTRs  should  be  clear,  concise,  and  should  contain 
all  required  information. 

Recommendations 

• The  CT  system  should  be  easy  to  use  and  learn.  At  the  time  of  building  the 
CT  system,  minimal  assumptions  about  the  CT  system  operator’s  skills 
should  be  made.  The  CT  system  operator  command  set  should  be  compact 


^SO  10303-34,  Working  draft  in  TCI  84/S C4/WG6,  "Conformance  testing  methodology  and 
framework:  abstract  test  methods." 
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and  relatively  simple  to  master  and  use.  Information,  error  messages,  and 
prompts  should  be  clear  and  easily  interpretable. 

The  CT  system  should  incorporate  tools  to  automate,  as  far  as  possible,  the 
entire  CT  process  thereby  minimizing  the  room  for  human  error. 

Comprehensive  documentation  should  be  available  for  installation, 
configuration  and  use  of  the  CT  system.  The  CTR  generators  should  follow 
the  format  requirements  of  ISO  10303-32  for  CTRs. 


3J,4  Equivalence  of  CT  Systems 

The  issue  of  CT-system  equivalence  assumes  importance  when  multiple  CT  systems 
for  the  same  set  of  application  protocols  have  to  be  evaluated.  The  primary  criterion 
is  to  show  comparability  of  conformance  test  results  between  CT  systems.  The  first 
step  towards  achieving  this  is  to  use  a common  ATS. 

While  it  is  not  possible  to  formally  prove  the  equivalence  of  two  CT  systems,  multiple 
ETSs  with  identical  SUTs  provide  a reasonable  level  of  confidence  that  future  lUT  CT 
results  would  be  comparable.  At  least  one  of  the  SUTs  has  to  be  non-conforming  to 
ensure  the  system  can  detect  problems  properly. 

Demonstrating  the  equivalence  of  CT  systems  with  a high  degree  of  confidence  is 
technically  challenging  and  demands  rigorous  examination  of  conformance  test  suite 
logs  and  results.  Identical  circumstances  have  to  be  maintained  on  each  CT  system 
throughout  the  evaluation  process  and  the  maximum  possible  amount  of  data  logged 
for  post-completion  evaluation. 

Recommendations 

• Where  a reference  implementation  capable  of  generating  known  errors  in  a 
controlled  manner  exists,  it  should  be  used  in  the  equivalence  evaluation 
process. 

• The  ETCs  should  be  carefully  planned,  run  in  identical  sequences,  and  the 
results  and  logs  subjected  to  meticulous  examination. 

• If  a CT  system  is  modified  (in  any  manner)  by  its  developer,  it  should  again 
be  subjected  to  the  same  process. 
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3.2  Executable  Test  Suite  Generation 


It  is  a matter  of  general  agreement  that  the  standards  committee  that  develops  a 
particular  AP  will  also  oversee  the  development  of  the  ATS  for  that  AP.  Once 
officially  accepted,  the  ATS  will  then  be  used  by  testing  laboratories  to  develop  ETSs. 
These  tasks  will  be  performed  in  accordance  with  the  ISO  TC184/SC4  10303-30  class 
of  standards  which  specify  different  abstract  test  methods,  development  of  ATSs,  and 
the  policies  and  procedures  for  the  development  of  ETSs  for  STEP.  That  an  ETS 
correctly  implements  its  parent  ATS  is  a matter  important  enough  to  warrant  attention 
by  the  Certification  Body. 


3.2.1  Policy  Issues 

The  need  for  openness  in  STEP  CT  systems  requires  the  availability  of  detailed 
information  on  the  CT  system,  ATSs,  and  ETS  tools.  Also,  the  need  for  strictly 
adhering  to  standards  requires  documents  and  forms  be  filled  out  before,  during,  and 
after  the  test  campaign.  It  is  the  mandate  of  the  CT  system  builder  to  provide 
sufficient  assurance  of  quality  and  coverage  in  areas  which  the  standard  specifies  as 
required. 

Recommendations 

• The  CT  system  builder  must  provide  all  documentation  for  the 
implementation  of  the  AP  to  be  tested.  Such  documentation  should  provide 
all  information  necessary  to  prepare  the  SUT  for  conformance  testing  and 
include  a description  of  the  conformance  assessment  process,  a definition  of 
the  CT  system  configuration,  and  a description  of  any  special  requirements 
placed  on  the  SUT  by  the  CT  system. 

• The  ETS  should  test  lUT  response  to  both  valid  and  invalid  behavior  by  the 
CT  system.  Invalid  behavior  should  include  illegal  parameters  and  invalid 
parameter  values. 

• The  conformance  testing  procedures  used  by  the  testing  laboratory  should 
be  made  public  to  help  ensure  that  the  conformance  assessment  process  is 
valid. 
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3,2,2  Abstract  and  Executable  Test  Suite  Coverage 

One  major  point  of  evaluation  of  the  CT  system  must  be  an  analysis  of  how  well  the 
CT  system  covers  the  ATS  (and  therefore  the  ETS)  of  the  AP  to  which  conformance 
is  being  tested.  Since  the  ATS  development  process  will  have  public  input,  most 
concerns  about  coverage  will  be  ironed  out  for  the  ATS  during  the  standardization 
process.  However,  deficiencies  could  be  identified  with  the  use  of  the  ETS  and 
improved  coverage  sought. 

Recommendation 

• An  AP  coverage  method  should  be  used  to  establish  how  well  an  ATS  covers 
its  associated  AP. 

Empirical  methods  of  measuring  AP  coverage  are  available.  One  such  method 
is  specified  in  the  ITI  technical  report  "Test  Coverage  Analysis  and  Measurement 
(TCAM):  A Practical  Approach  to  Determining  Coverage"  (document  ITI 
TR-87-14.1). 

3,3  CT  System  Builder 

The  organization  charged  with  building  a CT  system  has  to  satisfy  certain  important 
criteria:  a commitment  to  STEP  and  independence  from  all  individual  participants  in 
the  STEP  community.  Moreover,  the  CT  system  builder  must  also  make  specific 
commitment  to  maintain  the  CT  system  and  undertake  adequate  quality  assurance 
measures. 


3,3,1  Commitment 

The  CT  system  builder  must  indicate  its  long-term  commitment  to  STEP  and 
demonstrate  its  ability  to  support  the  CT  system.  Modifications  to  a CT  system  are  an 
ongoing  process  for  any  number  of  reasons.  The  CT  system  configuration  may  be 
impacted  by  new  versions  of  hardware  and  software;  although  the  CT  system  will  have 
undergone  evaluation  prior  to  release,  continual  debugging  is  often  necessary  as  more 
implementations  undergo  CT;  or  changes  to  an  AP  or  its  ATS  may  warrant 
modification  to  system  code.  If  the  CT  system  builder  chooses  not  to  continually 
maintain  the  CT  system,  a breakdown  in  the  CT  service  may  occur. 
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Recommendations 


• The  CT  system  builder  should  be  committed  to  keeping  the  CT  system 
up-to-date  with  the  base  standards  and  APs  as  they  change. 

• The  CT  system  builder  must  maintain  adequate  liaison  with  several 
organizations  to  keep  abreast  with  ongoing  issues. 

These  organizations  include:  standards  committees,  peer  conformance  testing 
organizations  abroad,  and  user/vendor  forums  and  consortia.  Maintaining  such 
contact  enables  the  CT  system  builder  to  anticipate  future  trends  and  needs,  and 
act  promptly  to  respond  to  them. 


3.3,2  Nonalignment 

The  product  developers  who  will  use  the  conformance  testing  service  must  be  assured 
that  the  CT  system  builder  is  an  independent  body  with  no  direct  commercial 
affiliations.  This  is  important  to  maintain  impartiality  of  the  CT  system  and  also  to 
encourage  the  developers  to  utilize  the  conformance  testing  service  to  improve  their 
products. 

Recommendation 

• An  independent  organization,  such  as  NIST,  should  be  charged  with  the 
responsibility  of  providing  the  STEP  CT  system  and  associated  tools.  In 
performing  this  task,  the  body  should  act  to  remain  nonaligned  and  make 
acquisition,  design,  and  implementation  decisions  in  keeping  with  this 
mandate. 


3.3.3  Maintenance,  Revision,  and  Quality  Assurance 

As  in  any  other  complex  project  involving  hardware  and  software,  the  CT  system 
builder  has  to  maintain  adequate  controls  on  the  revision  of  the  CT  system  and  on  error 
tracking  and  fixing.  Since  users  and  vendors  would  be  able  to  buy  or  lease  CT  systems 
for  in-house  use,  they  should  also  have  access  to  points  where  they  can  report 
problems. 
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Once  a CT  system  is  accepted  for  use,  there  has  to  be  a mechanism  for  ensuring  its 
continued  quality.  The  organization  that  chartered  the  CT  system  builder  should 
independendy  assess  the  quality  of  the  CT  system. 

The  CT  system  builder  should  be  able  to  respond  to  errata  and  other  changes  to  the 
standards  (APs).  Changes  in  the  CT  system  should  be  made  in  an  orderly  manner  and 
all  known  users  should  be  informed  of  new  CT  system  releases. 

Recommendations 

• The  revision  control  policies  and  schemes  for  the  CT  system  builder  should 
be  reviewed  by  user  and  vendor  groups  to  determine  if  they  achieve 
adequate  quality  control. 

• The  CT  system  builder  should  have  procedures  in  place  to  receive  problem 
reports  and  act  on  them  in  a timely  manner. 

If  the  problem  is  with  the  ATS  or  AP,  the  ISO  TC184/SC4  community  and  all 
known  users  should  be  notified  immediately  and  work  begun  to  fix  it.  If  the  CT 
system  is  the  problem,  work  to  rectify  it  should  be  started  and  users  notified.  All 
problems  and  fixes  should  be  logged. 

• The  CT  system  builder  should  have  procedures  in  place  for  tracking  errata 
and  other  changes  in  the  APs  and  determining  the  impact  of  these  changes 
on  the  CT  system  and  ATSs. 

If  changes  are  required,  the  nature  of  the  work,  expected  problems,  and  projected 
time  of  completion  should  be  made  known  to  all  users  of  the  CT  system. 

• CT  system  releases  should  be  accompanied  by  comprehensive  documentation 
including:  system  fixes  since  last  release,  AP  changes  affecting  the  new 
release,  and  remaining  known  problems. 

• The  organization  that  initiates  and  funds  developing  the  CT  system  must 
have  in  place  procedures  by  which  problems  and  complaints  could  be 
independently  accepted  and  progress  monitored.  Procedures  to  resolve  such 
problems  have  to  be  set  up  and  implemented. 
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The  CT  system  builder  should  consider,  and  CT  services  and  procedures 
reflect,  the  quality  assurance  requirements  identified  in  the  ISO  9000  series^ 
or  equivalent. 


3,4  The  CT  System  Evaluation  Process 

The  CT  system  builder  may  desire  to  maintain  a committee  (comprised  of 
representatives  from  CT  participating  organizations)  with  the  mandate  of  evaluating  all 
relevant  aspects  of  the  CT  system  and  the  CT  system  builder  (see  previous  sections). 
Only  after  satisfactory  completion  of  these  procedures  would  the  CT  system  builder 
claim  recognition  of  its  CT  system.  The  following  subsections  make  broad 
recommendations  on  CT  system  qualification.  If  CT  systems  are  to  be  developed  and 
put  in  place,  the  burden  of  evaluation  must  not  be  too  heavy. 


3,4,1  Evaluation  of  the  CT  System 

The  CT  system  builder  would  seek  acceptance  of  its  CT  system  through  the  committee 
mentioned  in  Section  3.4.  The  committee  would  evaluate  the  CT  system.  This 
evaluation  should  be  based  on  criteria  that  have  been  discussed  in  the  sections  above 
(repeatability,  comparability  of  test  results,  etc.).  All  evaluation  done  during  this 
process  would  use  ETS  tools  already  approved  by  the  committee. 

Throughout  the  evaluation  process,  the  committee  must  work  with  the  CT  system 
builder  to  resolve  any  problems  encountered  without  compromising  the  quality  and 
integrity  of  the  CT  system  itself.  Upon  acceptance  of  the  CT  system,  complete  and 
detailed  documentation  would  be  released  to  show  that  the  CT  system  meets  the  criteria 
for  recognition. 


^ISO  9000,  Quality  management  and  quality  assurance  standards  - guidelines  for  the  selection 
and  use. 

ISO  9001,  Quality  systems  - model  for  quality  assurance  in  design/development,  production, 
installation  and  servicing. 

ISO  9002,  Quality  system  - model  for  quality  assurance  in  production  and  installation. 

ISO  9003,  Quality  systems  - model  for  quality  assurance  in  final  inspection  and  test. 

ISO  9004,  Quality  management  and  quality  system  elements  - guidelines. 

[The  above  international  standards  are  available  through  the  ISO  Secretariat:  1,  rue  de  Varembe, 
Case  postale  56,  CH-1211  Geneva  20,  Switzerland.] 
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3,4,2  Recognition 


The  granting  of  recognition  would  be  done  for  a specific  CT  system  and  a specific  set 
of  APs,  the  versions  of  each  being  specified.  Recognition  would  be  valid  until 
specifically  withdrawn  by  the  committee  which  constantly  monitors  CT  activity. 
Withdrawal  of  recognition  would  occur  if  it  were  determined  that  the  CT  system 
builder  failed  to  show  commitment  to  maintain  the  CT  system  and/or  adequately 
perform  the  usual  tasks  of  bug  fixing,  upgrading,  and  user  support. 
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4.  Milestones 


The  previous  sections  in  this  report  emphasized  that  development  of  STEP  application 
protocols  and  associated  implementations  require  the  availability  of  conformance  testing 
capabilities  to  reap  the  benefits  of  standardization.  It  is  essential  to  have  in  place  a 
conformance  testing  capability  within  the  United  States  as  early  as  possible.  The 
United  States  should  thus  have  an  aggressive  STEP  CT  development  program  of  its 
own,  not  relying  on  other  countries.  The  benefits  are  many:  bringing  competitiveness 
to  the  industry,  nurturing  the  growth  of  expertise  at  home,  and  strengthening  the 
position  of  U.S.  members  in  international  standards  bodies. 

The  first  step  in  developing  STEP  CT  capabilities  in  the  United  States  is  to  come  up 
with  a plan  and  a realistic  schedule  for  CT  system  development  and  deployment.  Such 
a plan  should  be  in  line  with  the  needs  of  major  STEP  users  and  should  take  into 
account  the  current  state  of  the  standards  development  efforts.  This  section  enumerates 
a set  of  milestones  for  developing  and  deploying  a STEP  CT  system  and  starting  a CT 
service.  These  milestones  should  be  coordinated  with  the  NIST  Development  Plan  for 
STEP  CT  Services. * 

There  are  three  major  task  groups: 

• CT  System  Development 

• CT  System  Validation 

• Technology  Transfer  and  Testing  Laboratory  Accreditation 


4d  CT  System  Development 

This  subsection  details  the  milestones  for  the  task  of  CT  system  development. 


4,1.1  Generating  a Requirements  and  Development  Plan 

This  milestone  will  be  the  culmination  of  a requirements  analysis  task  that  will  focus 
on  three  important  issues  affecting  design  and  development  of  the  CT  system.  The 


^Kemmerer,  Sharon,  National  PDES  Testbed  Development  Plan,  STEP  Conformance  Testing 
Service.  NISTIR  4641,  August  1991. 
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milestone  will  be  the  generation  of  a high-level  requirements  document,  a development 
plan,  and  an  inter-organization  advisory  team.  Together  these  will  provide  important 
input  to  the  system  design  process  and  focus  on  building  the  capabilities  to  test 
conformance  to  the  selected  APs. 

First,  general  system  requirements  need  to  be  identified  to  feed  into  later  tasks.  This 
task  will  solicit  input  from  interested  parties  (users,  vendors,  standards  bodies)  on  what 
is  needed  in  a CT  system.  Early  involvement  in  the  CT  process  will  serve  to  unify  the 
participants’  view  and  offer  a forum  for  discussion.  Also,  users’  concerns  will  be  taken 
into  account  from  the  start.  This  task  will  be  done  by  requesting  information  from  all 
concerned,  and  by  hosting  one  or  more  workshops  to  discuss  and  evaluate  issues  and 
select  options. 

Second,  APs  have  to  be  chosen  for  the  CT  system.  This  choice  will  depend  on  the 
priorities  of  the  funding  agency  in  conjunction  with  prospective  users  of  the 
technology.  As  far  as  possible,  APs  that  have  advanced  to  or  are  close  to  becoming 
draft  international  standards  (DIS)  should  be  selected. 

Third,  the  CT  system  developer  should  invite  voluntary  participation  fi*om  interested 
organizations.  These  organizations  may  want  to  seek  assurance  that  work  is 
progressing  on  schedule  and  in  alignment  with  the  needs  of  the  user  and  vendor 
communities,  activities  of  the  standards  bodies,  and  international  STEP  activities. 


4.1.2  Accept  Abstract  Test  Cases 

It  has  been  recommended  that  standards  bodies/committees  developing  the  APs  bear 
the  responsibility  of  generating  ATSs.  This  task  will  examine  the  ATSs  thus  generated. 
Ambiguities  that  could  affect  ATS  interpretation  during  the  development  of  ETSs 
should  be  brought  to  the  attention  of  ISO  TC184/SC4  and  resolved  at  this  stage.  This 
examination  may  also  analyze  coverage  and  make  recommendations  to  the  appropriate 
committee  for  changes  to  improve  coverage  or  reduce  redundancy. 


4.1.3  Define  or  Accept  Abstract  Test  Methods 

This  task  will  accept  abstract  test  methods  for  CT  of  STEP-based  implementations. 
The  output  will  be  a document  describing,  in  general  terms,  the  abstract  test  methods 
that  will  be  used  for  the  selected  APs. 
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ISO  10303-34  of  STEP  is  concerned  with  defining  and  describing  abstract  test  methods 
which  take  into  account  the  control  and  observation  points  of  STEP  implementations. 
Since  this  CT  system  development  task  will  be  the  first  for  STEP,  it  may  well  be 
possible  that  for  the  APs  selected,  the  abstract  test  methods  in  STEP  Part  34  may  need 
to  be  modified  or  redefined. 


4,1,4  Generate  a Detailed  CT  System  Requirements  Specification 

This  task  will  develop  a requirements  specification  for  the  CT  system.  The  previous 
four  tasks  will  have  generated  implicit  and  explicit  requirements  for  the  CT  system. 

This  task  will  describe  the  needed  capabilities  of  the  CT  system.  Moreover,  the 
requirements  specification  should  be  flexible  enough  for  modification.  As  the  first  CT 
effort  for  STEP,  it  is  almost  certain  that  changes  will  be  required  as  work  on  STEP 
progresses  and  new  requirements  emerge. 


4,1,5  Develop  CT  System  Architecture  and  Design 

This  task  will  develop  a detailed  architectural  design  and  a detailed  internal  design. 
The  architectural  design  will  decompose  the  CT  system  into  system  modules,  interfaces, 
and  interconnections  and  provide  detailed  descriptions  of  these  items.  It  will  also 
specify  the  hardware  and  software  configuration  used  to  build  and  operate  the  CT 
system. 

The  detailed  internal  design  will  select  and/or  define  algorithms  and  data  structures  that 
will  be  used  to  build  the  CT  system  components  according  to  the  requirements 
specification.  This  task  will  also  consider  and  identify  existing  tools,  utilities,  and 
services  to  efficiently  implement  these  objects. 

The  plan  to  implement  the  CT  system  will  be  refined,  taking  into  account  evolving 
requirements  of  STEP  CT  as  STEP  becomes  standardized  and  priorities  change. 


4,1,6  Develop/Procure  Platform  and  Development  Tools 

The  output  of  this  task  will  be  a functional  hardware/software  platform  on  which 
development  can  begin  and  unit  testing  can  be  done.  As  the  development  of  the  CT 
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system  progresses,  more  equipment  and  tools  may  have  to  be  added  to  the  platform  to 
assess  the  capability  and  functionality  of  the  modules  developed.  A detailed  list  of  the 
kinds  of  tools  usually  needed  for  CT  can  be  found  in  Section  2.3.4. 


4dJ  Develop  Abstract  Test  Suites 

For  each  AP  selected,  an  ATS  is  necessary.  The  output  of  this  task  will  be  ATSs  for 
the  selected  APs  with  complete  and  comprehensive  documentation. 

An  ATS  will  have  to  be  accepted  and  supported  internationally  and  be  consistent  with 
worldwide  STEP  efforts.  Thus,  a procedure  to  process  existing  ATSs  through  ISO 
must  be  put  in  place. 

Some  of  the  ATSs  may  have  to  be  developed.  These  should  go  through  the  same 
acceptance  procedure  referred  to  above.  As  emphasized  before,  all  accepted  ATSs 
should  be  well  documented  and  internationally  standardized. 


4.1,8  Construct  and  Integrate  CT  System 

This  task  will  use  the  detailed  CT  system  architecture  and  design  as  input  to  build  the 
CT  system.  The  output  of  this  task  will  be  an  alpha  version  of  the  CT  system 
accompanied  by  comprehensive  documentation. 

This  is  a significant  development  component  of  the  CT  effort  and  will  incorporate  an 
ETC  generator  for  the  selected  APs;  compilers  and  parsers  for  the  STEP  description 
language,  ISO  CD  10303-11,  EXPRESS;  and  comprehensive  documentation  and  a 
capabilities  list  using  the  requirements  specifications  as  a baseline.  As  far  as  possible, 
the  components  of  the  CT  system  will  be  unit-tested  as  they  are  built.  However, 
substantial  system  testing  will  need  to  be  done  separately  as  described  in  the  following 
section. 
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42  CT  System  Validation 


After  completion  of  the  CT  system,  the  system  will  have  to  be  thoroughly  tested.  An 
undesirable  concept  is  to  build  a test  bed  that  is  bigger  than  the  CT  system  being 
evaluated.  Such  a process  could  spiral  out  of  control  and  make  the  whole  CT  concept 
collapse  upon  itself. 


422  Develop  CT  System  Validation  Plan 

Developing  a validation  plan  for  the  CT  system  will  be  necessary.  In  the  absence  of 
(certifiably)  conforming  product  implementations  that  could  be  used  to 
"reverse-validate"  the  CT  system  and  also  to  avoid  any  controversy  in  the  STEP 
community,  a limited,  independent  assessment  is  appropriate.  The  output  of  this  task 
will  be  a validation  process  and  an  implementation  plan  to  validate  the  fost  CT  system. 


422  Validate  CT  System 

This  task  will  use  an  independent  test  bed  to  validate  the  CT  system.  The  validation 
and  "fixing"  of  the  CT  system  will  be  done  iteratively:  problems  discovered  will  be 
fixed  and  the  CT  system  retested  starting  fi-om  a predetermined  baseline.  The  output 
of  this  task  will  be  a fully  validated  CT  system  ready  to  be  used  to  perform  CT  on 
product  implementations  based  on  the  selected  APs. 


43  Technology  Transfer  and  Testing  Laboratory  Accreditation 

The  STEP  CT  system  developed  in  the  above  tasks  will  represent  a major  step  toward 
building  a credible  capability  to  perform  CT  on  STEP-based  products.  This  CT 
capability  will  be  akin  to  developing  new  technology,  may  bring  with  it  new  testing 
approaches,  and  will  represent  a significant  advancement  in  the  area  of  standardized 
product  data  representation. 

To  disseminate  this  new  technology  and  leverage  its  utility  many  times  over,  it  has  to 
be  transferred  to  testing  laboratories  that  would  then  independently  perform  CT.  To 
begin  to  assess  the  quality  and  credibility  of  these  testing  laboratories,  one  may  choose 
to  ensure  the  testing  laboratories  chosen  for  the  task  have  a reputation  for  quality  work 
in  areas  related  to  STEP.  It  is  also  necessary  that  the  CT  technology  be  transferred  to 
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the  testing  laboratories  in  the  right  manner  and  that  the  testing  laboratories  possess  a 
certain  level  of  expertise  in  the  technology  area  in  which  they  are  offering  CT  services. 
Thus,  it  is  necessary  to  have  in  place  detailed  procedures  for  technology  transfer  and 
testing  laboratory  accreditation.  This  task  could  use  the  GOSIP  or  POSEX  accreditation 
methods  as  models. 


4,3d  Establish  a Conformance  Testing  Service 

The  first  task  is  to  start  a CT  service  at  the  site  where  the  CT  system  was  developed. 
This  would  assess  the  new  procedures  as  well  as  further  evaluate  the  CT  system  before 
release  to  other  sites.  Offering  this  service  requires  detailed  procedures  to  perform  CT 
along  with  forms  for  CT  reports  and  logs,  means  of  arbitration,  and  fee  structures.  This 
task  develops  a framework  for  offering  CT  services  including:  technical,  commercial, 
administrative,  and  legal  guidelines.  It  also  includes  developing  business  plans  which 
recognize  the  need  for  economically  viable  testing  laboratories,  advertising,  and 
initiating  STEP  CT  services. 


4,3,2  Develop  Technology  Transfer  Plan 

As  discussed  above,  technology  transfer  will  be  an  important  task  in  making  STEP  CT 
services  available.  This  task  develops  a plan  to  transfer  the  CT  technology  to  candidate 
testing  laboratories.  It  lays  out  the  requirements  for  staff,  facility  resources,  and 
licensing  of  tools  and  methods  used  in  CT  systems. 


4,3,3  Develop  Testing  Laboratory  Accreditation  Procedure 

This  task  develops  procedures  to: 

• Select  testing  laboratories  for  transfer  of  the  CT  technology  based  on  their 
technical,  commercial,  and  administrative  capabilities. 

• Assess  and  evaluate  the  testing  laboratories  on  the  basis  of  their  expertise  in 
STEP,  quality  and  experience  of  personnel,  adequate  equipment  and  quality 
control,  and  adherence  to  CT  procedures. 
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Too  much  bureaucracy  in  this  process  will  lead  to  high  costs,  a gatekeeper  mentality, 
and  ultimately,  failure  in  creating  a useful  CT  system. 


43,4  Invite  and  Approve  Candidate  Laboratories 

This  task  advertises  for  testing  laboratory  applicants.  It  will  evaluate  these  applicants 
on  the  basis  of  the  criteria  described  in  task  4.3.3  above.  The  output  of  this  task  is  a 
set  of  testing  laboratories  that  have  passed  the  selection  criteria. 


4,3,5  Start  Up  Conformance  Testing  Service 

This  task  transfers  the  CT  technology  to  the  selected  testing  laboratories.  The  output 
of  this  task  is  a register  of  testing  laboratories  with  a list  of  the  APs  for  which  they 
offer  CT  services. 


44 


5.  Summary  and  Conclusion 


Developing  STEP  conformance  testing  capability  will  be  an  important  task  in  the  near 
future.  This  report  discussed  the  important  issues  and  established  a baseline  for 
high-level  needs  and  requirements,  CT  system  evaluation  procedures,  and  milestones 
for  building  and  offering  a STEP  CT  system  in  the  United  States. 

The  issues  discussed  in  this  report  have  emphasized  the  following: 

• Appropriate  conformance  testing  is  an  economical  proposition  in  the  long 
run  and  should  be  pursued  actively  in  order  to  encourage  adoption  of  the 
standard. 

• Although  developing  and  establishing  facilities  for  CT  is  expensive,  the 
long-term  cost  of  not  doing  CT  is  even  more  expensive. 

• Conformance  testing  may  be  needed  for  implementations  that  utilize 
databases  of  product  data  as  well  as  exchange  implementations. 

• For  the  United  States  to  rely  on  other  countries  to  undertake  CT  would 
lessen  the  capability  of  American  companies  to  compete  in  the  international 
marketplace. 

• The  needs  of  the  STEP  product  vendor  community  must  be  addressed,  with 
open  CT  systems  being  an  important  contribution. 

• CT  software  tools  are  a very  important  component  of  CT  systems.  They 
automate  CT  tasks  and  help  develop  new  ATSs  and  supporting  ETSs. 

• Evaluation  of  CT  systems  and  services  is  an  important  issue.  Evaluation 
must  ensure  that  CT  is  accurate,  honest,  and  fair  while  not  being  an  undue 
burden  on  the  testing  laboratories. 

• The  CT  system  builder  must  be  committed,  nonaligned,  and  provide  quality 
assurance. 


The  discussions  in  this  report  led  to  the  following  conclusions: 
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The  United  States  should  initiate  the  building  of  a STEP  CT  system. 


The  STEP  CT  system  that  is  built  should  be  available  for  use  by  all 
interested  parties  at  a nominal  cost.  Easy  availability  will  help  produce 
better  products,  improve  the  level  of  vendors’  expertise,  and  build  user  and 
vendor  confidence. 

The  CT  system  that  is  built  should  be  highly  automated  and  also  include 
utilities  and  software  tools  to  automate,  to  a large  extent,  the  tasks  of 
interpreting  CT  results,  developing  ETSs,  and  generating  conformance  test 
reports. 

The  ISO  TC184/SC4  should  be  responsible  for  developing  and/or  ratifying 
ATSs  for  application  protocols. 

The  initial  CT  service  should  start  small  based  on  a limited  set  of  APs.  The 
CT  capability  should  expand  as  needed. 

There  should  be  a process  to  address  the  needs  of  vendors  and  users  before 
and  during  the  design  and  development  of  the  CT  system.  The  main  goal 
is  to  serve  the  users. 
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6.  Glossary 


abstract  test  case  (ATC):  One  or  more  files,  encapsulating  the  test  purpose,  which 
provide  the  basis  from  which  parameterized  executable  test  cases  are  derived. 

abstract  test  method:  The  description  of  how  an  lUT  is  to  be  tested,  given  at  the 
appropriate  level  of  abstraction  to  make  the  description  independent  of  any  particular 
implementation  of  testing  tools  or  procedures,  but  with  enough  detail  to  enable 
executable  test  cases  to  be  generated  from  this  test  method. 

(laboratory)  accreditation:  The  formalized  initial  and  continuing  process  of  ensuring 
a testing  laboratory  is  competent  to  carry  out  specific  (types  of)  tests. 

NOTE  - The  term  "laboratory  accreditation"  covers  the  recognition  of  both  the 
technical  competence  and  the  impartiality  of  a testing  laboratory.  Accreditation 
is  normally  awarded  following  successful  laboratory  assessment  and  is  followed 
by  appropriate  surveillance. 

accreditation  body:  A body  that  conducts  and  administers  a laboratory  accreditation 
scheme  and  grants  accreditation. 

NOTE  - An  accreditation  body  may  wish  to  delegate  fully  or  partially  the 
assessment  of  a testing  laboratory  to  another  competent  body  (assessment  agency). 
While  it  is  recognized  that  this  may  be  a practical  solution  to  extending 
recognition  of  testing  laboratories,  it  is  essential  that  such  assessment  be 
equivalent  to  that  applied  by  the  accreditation  body  and  that  the  accreditation 
body  take  full  responsibility  for  such  extended  accreditation. 

certificate  of  conformity,  certificate  of  conformance:  A document  issued  under  the 
rules  of  a certification  system  indicating  that  adequate  confidence  is  provided  that  an 
lUT  is  in  conformity  with  a specific  standard  or  technical  specification  as  determined 
through  use  of  a specified  abstract  test  method. 

certification  body:  An  impartial  body  possessing  the  necessary  competence  and 
reliability  to  operate  a certification  system,  and  in  which  the  interests  of  all  parties 
concerned  with  the  function  of  the  system  are  represented. 

client  (of  a testing  laboratory):  The  organization  that  submits  an  implementation  for 
conformance  testing. 
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comparability  (of  results):  Characteristic  of  conformance  assessment  processes,  such 
that  execution  on  the  same  SUT,  in  different  testing  laboratories,  leads  to  the  same 
overall  summary. 

conformance  assessment  process:  The  complete  process  of  accomplishing  all 
conformance  testing  activities  necessary  to  determine  the  conformance  of  an 
implementation  to  an  application  protocol. 

conformance  test  report:  A document  written  at  the  end  of  the  conformance 
assessment  process,  which  provides  both  summary  and  detailed  information.  The  first 
part  gives  the  overall  summary  of  the  conformance  of  the  lUT  to  the  standard  for 
which  conformance  testing  was  carried  out;  the  second  gives  the  details  of  the  testing 
carried  out  for  a particular  standard. 

conformance  testing:  The  testing  of  a candidate  product  for  the  existence  of  specific 
characteristics  required  by  a standard;  testing  the  extent  to  which  a particular 
implementation  under  test  is  a conforming  implementation,  (source:  ISO  Committee 
Draft  10303-31,  "Conformance  testing  methodology  and  framework:  general  concepts," 
May  20,  1992.) 

executable  test  case  (ETC):  A realization  of  an  abstract  test  case.  (The  form  of  the 
realization  is  still  under  development.) 

GOSIP:  Government  Open  Systems  Interconnection  Profile. 

Implementation  Under  Test  (lUT):  That  part  of  a product  which  is  to  be  studied 
under  testing,  which  should  be  an  implementation  of  one  or  more  characteristics  of  the 
standard(s). 

interoperability  testing:  Related  to  acceptance  testing,  but  specifically  applied  to  the 
examination  of  the  information  exchange  between  two  specific  lUTs  and  the  ability  of 
each  lUT  to  use  such  information. 

NOTE  - This  does  not  form  part  of  conformance  testing. 

lUT:  See  Implementation  Under  Test. 

POSIX:  Portable  Operating  System  Interface  for  Computer  Environments. 


48 


repeatability  (of  results):  Characteristic  of  a test  case,  such  that  repeated  executions 
on  the  same  SUT  under  the  same  conditions  lead  to  the  same  test  verdict,  and  by 
extension  a characteristic  of  a test  suite. 

SUT:  See  System  Under  Test. 

System  Under  Test  (SUT):  The  computer  hardware,  software,  and  communication 
network  required  to  support  the  lUT. 

test  campaign:  The  process  of  executing  the  (parameterized)  executable  test  suite  for 
a particular  lUT. 

test  report:  See  conformance  test  report. 

test  suite:  A complete  set  of  test  cases,  possibly  combined  into  nested  test  groups,  that 
is  necessary  to  perform  conformance  testing  for  a standard  or  group  of  standards. 

Note:  This  term  is  deprecated  unless  prefixed  by  abstract  or  executable. 

testing  laboratory:  An  organization  that  carries  out  the  conformance  assessment 
process.  This  can  be  a third  party,  a user  organization,  or  an  identifiable  part  of  the 
supplier  organization. 
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